Oct 31 03:16:53.718505 ip-10-0-143-24 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 03:16:53.718648 ip-10-0-143-24 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 03:16:53.718742 ip-10-0-143-24 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 03:16:53.719163 ip-10-0-143-24 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 03:17:04.936446 ip-10-0-143-24 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 03:17:04.936463 ip-10-0-143-24 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 88ca29bc50314ac58ba77146d4790799 -- Oct 31 03:18:52.157410 ip-10-0-143-24 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:18:52.586453 ip-10-0-143-24 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:18:52.586453 ip-10-0-143-24 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:18:52.586453 ip-10-0-143-24 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:18:52.586453 ip-10-0-143-24 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:18:52.593216 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593061 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.593216 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593212 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.593216 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593216 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.593216 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593220 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.593216 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593224 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.593216 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593228 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593232 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593238 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593241 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593245 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593248 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593253 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593256 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593260 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593263 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593267 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593271 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593275 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593279 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593285 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593289 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593292 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593296 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593300 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593304 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593308 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.593582 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593311 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593315 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593319 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593323 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593326 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593330 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593334 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593338 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593341 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593346 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593349 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593353 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593357 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593362 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593366 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593370 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593374 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593378 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593383 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593388 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.594456 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593392 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593397 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593401 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593405 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593409 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593413 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593418 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593422 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593427 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593432 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593436 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593440 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593444 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593448 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593453 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593457 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593461 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593465 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593469 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.595336 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593477 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593485 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593490 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593494 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593500 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593507 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593514 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593520 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593524 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593529 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593533 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593537 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593541 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593545 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593549 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593553 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593558 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593562 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.596160 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593566 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593570 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593580 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593585 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593589 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593593 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593597 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593615 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593619 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593624 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593628 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593632 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593636 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593640 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593644 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593649 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593653 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593657 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593661 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593666 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.596912 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593670 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593675 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.593679 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595105 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595117 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595122 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595127 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595130 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595135 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595141 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595145 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595149 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595154 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595158 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595162 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595167 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595172 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595177 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595181 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595186 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.597518 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595190 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595195 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595199 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595203 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595207 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595211 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595215 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595219 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595224 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595229 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595233 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595239 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595244 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595248 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595253 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595257 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595261 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595266 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595270 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.598000 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595274 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595279 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595283 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595289 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595293 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595297 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595302 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595306 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595310 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595315 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595320 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595324 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595328 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595332 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595336 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595340 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595344 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595349 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595353 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.598480 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595357 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595361 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595365 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595369 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595374 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595379 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595384 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595388 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595393 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595398 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595403 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595407 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595411 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595415 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595419 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595423 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595428 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595432 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595436 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.598935 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595440 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595446 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595453 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595459 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595465 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595471 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595475 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595479 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595484 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595488 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595493 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595497 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595501 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595506 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595512 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595519 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595523 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595528 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.599738 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595533 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595538 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595542 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595546 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595551 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595555 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595559 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595563 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595568 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595574 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595579 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595583 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595587 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.595592 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596521 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596534 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596539 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596543 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596549 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596553 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.600552 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596558 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596562 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596567 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596571 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596575 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596579 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596584 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596588 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596593 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596597 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596602 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596607 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596614 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596620 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596624 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596629 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596634 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596638 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596642 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.601282 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596646 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596651 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596655 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596659 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596663 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596668 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596672 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596676 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596681 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596685 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596689 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596693 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596698 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596703 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596706 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596711 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596715 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596719 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596724 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596728 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.601798 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596732 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596736 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596741 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596744 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596749 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596753 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596757 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596761 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596766 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596770 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596774 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596778 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596783 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596787 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596791 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596795 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596800 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596804 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596808 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596812 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.602294 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596816 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596820 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596825 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596831 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596834 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596839 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596844 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596848 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596852 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596864 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596868 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596872 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596876 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596880 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596885 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596889 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596893 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596897 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596901 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.602854 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596905 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596912 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596920 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596924 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596928 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596932 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596936 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596940 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596944 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596948 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596952 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596957 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596961 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596967 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596972 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596976 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596980 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596984 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596988 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.603345 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596993 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.596997 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.597002 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599151 2569 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599171 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599182 2569 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599189 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599196 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599201 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599208 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599216 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599221 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599226 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599232 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599237 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599243 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599248 2569 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599253 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599258 2569 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599263 2569 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599268 2569 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599279 2569 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599283 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:18:52.603788 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599289 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599294 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599301 2569 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599308 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599313 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599318 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599323 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599328 2569 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599333 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599338 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599344 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599350 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599357 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599362 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599367 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599372 2569 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599376 2569 flags.go:64] FLAG: --enable-server="true" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599381 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599388 2569 flags.go:64] FLAG: --event-burst="100" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599393 2569 flags.go:64] FLAG: --event-qps="50" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599398 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599403 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599408 2569 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599415 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:18:52.604339 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599420 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599425 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599430 2569 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599435 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599440 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599444 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599449 2569 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599454 2569 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599458 2569 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599463 2569 flags.go:64] FLAG: --feature-gates="" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599470 2569 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599475 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599480 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599485 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599490 2569 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599495 2569 flags.go:64] FLAG: --help="false" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599500 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599506 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599511 2569 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599517 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599523 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599528 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599533 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:18:52.604949 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599538 2569 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599542 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599547 2569 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599552 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599557 2569 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599562 2569 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599567 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599572 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599576 2569 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599582 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599586 2569 flags.go:64] FLAG: --lock-file="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599590 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599595 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599600 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599609 2569 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599614 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599619 2569 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599624 2569 flags.go:64] FLAG: --logging-format="text" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599629 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599634 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599639 2569 flags.go:64] FLAG: --manifest-url="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599643 2569 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599655 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599660 2569 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599667 2569 flags.go:64] FLAG: --max-pods="110" Oct 31 03:18:52.605537 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599672 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599677 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599681 2569 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599686 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599692 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599697 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599702 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599714 2569 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599719 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599724 2569 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599729 2569 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599734 2569 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599738 2569 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599742 2569 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599747 2569 flags.go:64] FLAG: --port="10250" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599751 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599756 2569 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0eb8281b2cccee03e" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599760 2569 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599765 2569 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599771 2569 flags.go:64] FLAG: --register-node="true" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599776 2569 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599782 2569 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599787 2569 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599791 2569 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599796 2569 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599802 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:18:52.606160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599807 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599812 2569 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599817 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599822 2569 flags.go:64] FLAG: --runonce="false" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599827 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599832 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599837 2569 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599841 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599846 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599854 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599859 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599869 2569 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599874 2569 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599879 2569 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599883 2569 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599888 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599892 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599897 2569 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599902 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599910 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599915 2569 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.599920 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601171 2569 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601220 2569 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601228 2569 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:18:52.606771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601234 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601241 2569 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601246 2569 flags.go:64] FLAG: --v="2" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601256 2569 flags.go:64] FLAG: --version="false" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601263 2569 flags.go:64] FLAG: --vmodule="" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601276 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.601282 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601738 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601748 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601751 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601755 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601757 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601760 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601762 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601765 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601768 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601770 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601773 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601776 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601779 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601782 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.607402 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601787 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601790 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601793 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601797 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601800 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601803 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601806 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601809 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601812 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601814 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601817 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601820 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601823 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601825 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601828 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601830 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601833 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601836 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601844 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601847 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.607905 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601849 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601853 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601855 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601858 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601861 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601863 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601866 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601868 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601871 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601873 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601876 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601878 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601881 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601883 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601886 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601889 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601891 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601894 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601897 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601900 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.608412 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601903 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601905 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601908 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601910 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601913 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601915 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601919 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601923 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601926 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601928 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601931 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601933 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601936 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601938 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601941 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601943 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601946 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601948 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601951 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.608876 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601953 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601956 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601959 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601961 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601964 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601967 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601969 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601972 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601974 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601977 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601979 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601982 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601985 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601989 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601993 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.601996 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602000 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602003 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602005 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.609391 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602008 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602011 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602013 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602016 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602018 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602021 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602024 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602026 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602029 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602031 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602034 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602036 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602039 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602041 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.602074 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602133 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.609848 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602138 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602141 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602145 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602147 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602150 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602152 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602155 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602157 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602160 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602163 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602167 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602171 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602174 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602177 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602180 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602183 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602185 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602188 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602190 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.610320 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602193 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602196 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602198 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602201 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602203 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602206 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602208 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602211 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602213 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602216 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602218 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602221 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602224 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602227 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602230 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602232 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602234 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602237 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602239 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602242 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.610767 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602244 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602247 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602249 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602252 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602255 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602258 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602260 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602263 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602265 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602268 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602270 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602273 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602275 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602277 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602280 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602282 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602285 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602287 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602289 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.611275 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602292 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602294 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602300 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602302 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602305 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602308 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602311 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602314 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602316 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602319 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602321 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602324 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602326 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602329 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602331 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602335 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602338 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602340 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602343 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602345 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.611763 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602348 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602350 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602353 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602355 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602358 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602360 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602363 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602365 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602367 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602370 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602372 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602374 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602378 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602382 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602386 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602390 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602392 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602395 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602397 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602400 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.612250 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602403 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602405 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602408 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602410 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602413 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602417 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.602419 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.609676 2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.609697 2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.609717 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.609722 2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609785 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609790 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609794 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609796 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609799 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609802 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609805 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609808 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.612726 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609811 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609814 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609817 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609819 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609822 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609824 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609827 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609830 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609832 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609835 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609838 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609841 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609843 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609846 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609848 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609851 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609854 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609857 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609860 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609862 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.613207 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609865 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609869 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609872 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609874 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609877 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609880 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609882 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609885 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609888 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609890 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609893 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609897 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609901 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609904 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609907 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609909 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609912 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609914 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609917 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.613683 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609919 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609922 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609925 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609928 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609932 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609936 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609939 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609942 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609944 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609946 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609949 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609952 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609954 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609958 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609960 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609964 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609966 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609969 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609971 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609974 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.614165 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609976 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609979 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609981 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609983 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609986 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609988 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609991 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609993 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609996 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.609998 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610000 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610003 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610005 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610008 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610010 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610014 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610016 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610018 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610021 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610023 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.614674 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610026 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610028 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610031 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610033 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610036 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610039 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610041 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610044 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610047 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610050 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610052 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610054 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610057 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610059 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610062 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610064 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610066 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610069 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.615185 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610071 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.610118 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610272 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610276 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610280 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610283 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610286 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610289 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610291 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610294 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610297 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610299 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610302 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610304 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610307 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610309 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.615623 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610312 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610315 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610318 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610320 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610323 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610326 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610329 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610332 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610335 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610337 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610340 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610342 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610345 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610347 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610350 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610352 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610354 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610357 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610359 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.616016 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610362 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610364 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610367 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610369 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610371 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610374 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610376 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610379 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610382 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610384 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610387 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610389 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610392 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610394 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610397 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610399 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610402 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610404 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610407 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.616461 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610410 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610412 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610415 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610418 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610420 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610423 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610425 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610428 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610430 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610433 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610435 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610438 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610440 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610443 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610445 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610448 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610450 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610454 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610457 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.616925 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610460 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610463 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610466 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610469 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610471 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610474 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610476 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610479 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610481 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610484 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610486 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610489 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610491 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610494 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610496 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610498 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610501 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610504 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610506 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610509 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.617457 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610511 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610514 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610516 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610519 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610522 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610524 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610526 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610529 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610531 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610534 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610536 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610540 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610544 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610547 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610550 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.610573 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:18:52.617915 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610605 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610609 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610612 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610615 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610617 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610620 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610623 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610625 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610628 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610630 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610633 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610635 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610638 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610642 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610646 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610649 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610652 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610654 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610657 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:18:52.618321 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610659 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610662 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610664 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610667 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610669 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610672 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610675 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610677 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610679 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610685 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610688 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610690 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610693 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610695 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610698 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610700 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610703 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610705 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610708 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610710 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:18:52.618777 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610712 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610715 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610717 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610720 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610722 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610724 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610727 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610734 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610737 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610740 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610742 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610745 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610748 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610751 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610753 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610756 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610758 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610761 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610763 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610766 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:18:52.619260 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610769 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610772 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610775 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610778 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610781 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610783 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610786 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610789 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610791 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610793 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610796 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610799 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610801 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610804 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610806 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610809 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610811 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610813 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610816 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:18:52.619709 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610820 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610823 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610826 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610828 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610831 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610833 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610836 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610838 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610849 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610853 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610856 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610858 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610861 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610863 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610867 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610870 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610873 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610876 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610878 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610881 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:18:52.620182 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610883 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610886 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610889 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610892 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610895 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610899 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610901 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:52.610904 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.610972 2569 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.613198 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.614074 2569 server.go:1014] "Starting client certificate rotation" Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.614176 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.614213 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.616110 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:18:52.620650 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.617246 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:18:52.627187 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.627167 2569 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:18:52.639122 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.639103 2569 log.go:25] "Validated CRI v1 image API" Oct 31 03:18:52.640029 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.640003 2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:18:52.644079 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.644048 2569 fs.go:135] Filesystem UUIDs: map[56076a89-e5fc-4be6-b5b3-b849a6c05132:/dev/nvme0n1p4 629f963b-e4a2-47be-be6f-f79868eed778:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 03:18:52.644176 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.644076 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 03:18:52.650229 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.650112 2569 manager.go:217] Machine: {Timestamp:2026-10-31 03:18:52.648651053 +0000 UTC m=+0.362610349 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120437 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282078128c76aaecef6ff181b05792 SystemUUID:ec282078-128c-76aa-ecef-6ff181b05792 BootID:88ca29bc-5031-4ac5-8ba7-7146d4790799 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:69:01:55:e6:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:69:01:55:e6:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:d4:8c:89:d3:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:18:52.650229 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.650231 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:18:52.650367 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.650322 2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:18:52.653694 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.653666 2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:18:52.653842 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.653696 2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-24.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:18:52.653890 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.653845 2569 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:18:52.653890 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.653852 2569 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:18:52.653943 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.653890 2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:18:52.653943 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.653908 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:18:52.655208 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.655197 2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:18:52.659462 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.659449 2569 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:18:52.659604 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.659594 2569 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:18:52.660416 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.660405 2569 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:18:52.660458 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.660421 2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:18:52.660458 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.660435 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:18:52.660458 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.660453 2569 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:18:52.660580 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.660462 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:18:52.660823 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.660808 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:18:52.663773 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.663751 2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:18:52.669953 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.669938 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:18:52.670903 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.670890 2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:18:52.670941 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.670922 2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:18:52.671103 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671078 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:18:52.671137 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671106 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:18:52.671137 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671113 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:18:52.671137 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671119 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:18:52.671137 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671125 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:18:52.671137 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671130 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:18:52.671786 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671778 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:18:52.671814 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671786 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:18:52.671840 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671822 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:18:52.671840 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671829 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:18:52.671840 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671838 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:18:52.671990 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.671847 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:18:52.672550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.672541 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:18:52.672581 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.672551 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:18:52.676568 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.676555 2569 server.go:1263] "Started kubelet" Oct 31 03:18:52.676978 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.676649 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:18:52.676978 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.676693 2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:18:52.676978 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.676758 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:18:52.677554 ip-10-0-143-24 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:18:52.678341 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.678324 2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:18:52.680293 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.680268 2569 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:18:52.683320 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.683298 2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 03:18:52.687079 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.687037 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:18:52.688363 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.688343 2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:18:52.688630 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.688608 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:52.689888 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.689871 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:18:52.690182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.690153 2569 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:18:52.690182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.690177 2569 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:18:52.690626 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.690591 2569 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:18:52.690626 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.690624 2569 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:18:52.692664 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692646 2569 factory.go:55] Registering systemd factory Oct 31 03:18:52.692748 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692667 2569 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:18:52.692882 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692871 2569 factory.go:153] Registering CRI-O factory Oct 31 03:18:52.692882 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692883 2569 factory.go:223] Registration of the crio container factory successfully Oct 31 03:18:52.692958 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692944 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:18:52.693014 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692967 2569 factory.go:103] Registering Raw factory Oct 31 03:18:52.693014 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.692982 2569 manager.go:1199] Started watching for new ooms in manager Oct 31 03:18:52.694153 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.694137 2569 manager.go:319] Starting recovery of all containers Oct 31 03:18:52.704766 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.704751 2569 manager.go:324] Recovery completed Oct 31 03:18:52.708576 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.708561 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.710343 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.710322 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.710399 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.710363 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.710399 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.710374 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.711001 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.710986 2569 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:18:52.711001 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.711001 2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:18:52.711124 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.711007 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.711124 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.711021 2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:18:52.711806 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.711793 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.711842 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.711817 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.711842 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.711826 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.713286 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.713269 2569 policy_none.go:50] "Start" Oct 31 03:18:52.713346 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.713299 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:18:52.713346 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.713313 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:18:52.715228 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.715213 2569 policy_none.go:44] "Start" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.755851 2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761080 2569 manager.go:341] "Starting Device Plugin manager" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.761142 2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761153 2569 server.go:87] "Starting device plugin registration server" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761420 2569 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761433 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761627 2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761734 2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.761747 2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.762321 2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.762366 2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.768262 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.769997 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.770027 2569 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.770083 2569 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:18:52.773135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.770186 2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 03:18:52.821701 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.821623 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-24.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 03:18:52.823664 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.821859 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-24.us-west-1.compute.internal.18e37f04356d1863 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-24.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-24.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:52.676528227 +0000 UTC m=+0.390487523,LastTimestamp:2026-10-31 03:18:52.676528227 +0000 UTC m=+0.390487523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-24.us-west-1.compute.internal,}" Oct 31 03:18:52.862533 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.862494 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.863551 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.863527 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.863698 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.863570 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.863698 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.863586 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.863698 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.863621 2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.870832 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.870806 2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal"] Oct 31 03:18:52.870927 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.870893 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.871862 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.871841 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.871945 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.871883 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.871945 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.871896 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.874163 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.874149 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.875052 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.875038 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.875122 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.875068 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.875122 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.875120 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.875395 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.875378 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.875431 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.875414 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.876252 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.876226 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.876252 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.876243 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-24.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 03:18:52.876398 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.876258 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.876398 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.876228 2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-24.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 03:18:52.876398 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.876272 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.876398 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.876220 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-24.us-west-1.compute.internal.18e37f0437711d02 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-24.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-24.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-24.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:52.710345986 +0000 UTC m=+0.424305281,LastTimestamp:2026-10-31 03:18:52.710345986 +0000 UTC m=+0.424305281,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-24.us-west-1.compute.internal,}" Oct 31 03:18:52.876620 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.876480 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 03:18:52.877366 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.877347 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.877443 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.877384 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:52.878115 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.878080 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:52.878191 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.878124 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:52.878191 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.878135 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:52.891288 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.891261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.891380 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.891300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.891380 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.891325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb3780775d238ba0d694f1e9cc7fb2a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"abb3780775d238ba0d694f1e9cc7fb2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.894700 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.894684 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:18:52.894756 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.894749 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b8kwl" Oct 31 03:18:52.908151 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.908130 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.911021 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.911000 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.981001 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.980964 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:52.981583 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.981549 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:52.989657 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.989634 2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.989657 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:52.989661 2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-24.us-west-1.compute.internal\": node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:52.991701 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.991686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.991756 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.991710 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb3780775d238ba0d694f1e9cc7fb2a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"abb3780775d238ba0d694f1e9cc7fb2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.991756 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.991728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.991845 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.991790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.991845 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.991812 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb3780775d238ba0d694f1e9cc7fb2a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"abb3780775d238ba0d694f1e9cc7fb2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:52.991845 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:52.991790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:53.112045 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.112011 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.213029 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.212978 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.213029 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.213010 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:53.216187 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.216168 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:53.313703 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.313645 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.414539 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.414468 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.515197 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.515158 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.614021 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.613988 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 03:18:53.614531 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.614191 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:18:53.614531 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.614190 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:18:53.614531 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.614249 2569 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3bc042c2ecd546b6b1e906ab067c5a8-f96d67daba3dd493.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-24.us-west-1.compute.internal.18e37f0437711d02\": read tcp 10.0.143.24:51668->3.18.80.101:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-24.us-west-1.compute.internal.18e37f0437711d02 default 4893 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-24.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-24.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-24.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:52 +0000 UTC,LastTimestamp:2026-10-31 03:18:52.871863257 +0000 UTC m=+0.585822560,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-24.us-west-1.compute.internal,}" Oct 31 03:18:53.616133 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.616115 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.717080 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.717005 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.817699 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.817656 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.899861 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.899815 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:13:52 +0000 UTC" deadline="2028-07-19 09:57:52.27056915 +0000 UTC" Oct 31 03:18:53.899861 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.899846 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15054h38m58.370726158s" Oct 31 03:18:53.918031 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:53.917990 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:53.943057 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.943030 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:18:53.944719 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.944702 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:18:53.944815 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.944734 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:18:53.944815 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:53.944762 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:18:54.019075 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.018998 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:54.084511 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.084477 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:54.120166 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.120137 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:18:54.180716 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.180682 2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:54.189764 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.189741 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:54.252223 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.252195 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:18:54.253195 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.253180 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:18:54.308129 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.308110 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:18:54.318157 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:54.318120 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb3780775d238ba0d694f1e9cc7fb2a.slice/crio-ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0 WatchSource:0}: Error finding container ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0: Status 404 returned error can't find the container with id ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0 Oct 31 03:18:54.318377 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:54.318356 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd30d86732a0a5d7b8493f1d7958e8b.slice/crio-0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee WatchSource:0}: Error finding container 0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee: Status 404 returned error can't find the container with id 0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee Oct 31 03:18:54.324880 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.323269 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:18:54.661654 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.661621 2569 apiserver.go:52] "Watching apiserver" Oct 31 03:18:54.687803 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.687773 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 03:18:54.757808 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.757769 2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:18:54.758160 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.758138 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-x75qg","openshift-network-operator/iptables-alerter-cnthq","kube-system/global-pull-secret-syncer-jvb8m","kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-ns5dr","openshift-multus/multus-additional-cni-plugins-fpp2c","openshift-multus/multus-r7w79","kube-system/konnectivity-agent-kms5j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79","openshift-cluster-node-tuning-operator/tuned-bljwr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-h4tqk","openshift-dns/node-resolver-vl82x","openshift-image-registry/node-ca-ftcg9"] Oct 31 03:18:54.761730 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.761707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.763652 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.763630 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.765486 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.765469 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.765553 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.765536 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:18:54.767397 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.767383 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.769541 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.769522 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.772575 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.772557 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.774766 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.774748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.776924 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.776907 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.779327 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.779309 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.781619 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.781600 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:54.781707 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.781688 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:18:54.783660 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.783645 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.785724 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.785688 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"0cd30d86732a0a5d7b8493f1d7958e8b","Type":"ContainerStarted","Data":"0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee"} Oct 31 03:18:54.785777 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.785768 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:54.785833 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.785819 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:18:54.788353 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.788335 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"abb3780775d238ba0d694f1e9cc7fb2a","Type":"ContainerStarted","Data":"ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0"} Oct 31 03:18:54.791159 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.791141 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:18:54.803406 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-kubernetes\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.803524 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-cni-binary-copy\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.803524 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-netns\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.803524 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-node-log\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.803524 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-lib-modules\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-device-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-conf-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803670 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l97\" (UniqueName: \"kubernetes.io/projected/20d028be-4c41-438b-86ac-2e2eadce25df-kube-api-access-p4l97\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-ovn\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.803713 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-host\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-tuned\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56e24d88-5120-43d1-9649-9245d1967788-iptables-alerter-script\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-var-lib-kubelet\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803886 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-registration-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803918 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-multus-daemon-config\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cnibin\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.803998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrk8\" (UniqueName: \"kubernetes.io/projected/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-kube-api-access-xzrk8\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.804121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-run\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-kubelet-config\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-sys-fs\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804211 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-system-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-env-overrides\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvhk\" (UniqueName: \"kubernetes.io/projected/0c92072e-6213-4831-b222-039305a4d122-kube-api-access-snvhk\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-modprobe-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-k8s-cni-cncf-io\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-multus-certs\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-etc-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-log-socket\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804531 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-dbus\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804562 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-sys\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-bin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-multus\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.804670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-kubelet\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-slash\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-conf\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804803 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rj4z\" (UniqueName: \"kubernetes.io/projected/c436b9f3-e0e4-4662-98bf-a140ef5c010c-kube-api-access-5rj4z\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-systemd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804837 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-var-lib-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-config\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c92072e-6213-4831-b222-039305a4d122-ovn-node-metrics-cert\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae4c6a55-1343-49ab-b514-78564e0f1d75-agent-certs\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysconfig\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.804987 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zjq\" (UniqueName: \"kubernetes.io/projected/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-kube-api-access-f7zjq\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8hv\" (UniqueName: \"kubernetes.io/projected/9de23b18-25f8-43c3-9463-152099741b6b-kube-api-access-zd8hv\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-system-cni-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ttk\" (UniqueName: \"kubernetes.io/projected/de3f9807-0638-4b57-b00a-bb9bc112505b-kube-api-access-t7ttk\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:54.805247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436b9f3-e0e4-4662-98bf-a140ef5c010c-tmp-dir\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.805891 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-kubelet\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.805891 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8th4\" (UniqueName: \"kubernetes.io/projected/56e24d88-5120-43d1-9649-9245d1967788-kube-api-access-f8th4\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.805891 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.805891 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805716 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.805891 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-hostroot\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.805891 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.805805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-etc-kubernetes\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.806239 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56e24d88-5120-43d1-9649-9245d1967788-host-slash\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.806239 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9de23b18-25f8-43c3-9463-152099741b6b-serviceca\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.806239 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-systemd\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c436b9f3-e0e4-4662-98bf-a140ef5c010c-hosts-file\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de23b18-25f8-43c3-9463-152099741b6b-host\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-systemd-units\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-tmp\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-cnibin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-socket-dir-parent\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-netns\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-bin\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806680 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-socket-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfg98\" (UniqueName: \"kubernetes.io/projected/449c2d15-e376-4ba5-a668-9c37ea713e3f-kube-api-access-gfg98\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-netd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806832 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-script-lib\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.808265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.808708 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806917 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.808708 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.806950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c6a55-1343-49ab-b514-78564e0f1d75-konnectivity-ca\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.810765 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.810735 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.810862 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.810787 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:54.810862 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.810801 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.810959 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.810871 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4wd7m\"" Oct 31 03:18:54.811979 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.811965 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz7gs\"" Oct 31 03:18:54.812076 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.811980 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.812185 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.812173 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.815670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.815652 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.815670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.815655 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:18:54.815911 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.815900 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.816053 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.816036 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltpjd\"" Oct 31 03:18:54.854677 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.854645 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:18:54.854859 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.854726 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tkfpd" Oct 31 03:18:54.903450 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903383 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:18:54.903450 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903400 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.903450 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903399 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.903450 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903411 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.903804 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903640 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:18:54.903850 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903831 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.903850 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903845 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:18:54.904011 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.903973 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:18:54.904011 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904001 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904130 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904151 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4ln2\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904161 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904171 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904152 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9x6rv\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904192 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904175 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:18:54.904182 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904174 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904205 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4kwqf\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904154 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xmnbh\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904218 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904274 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67ksm\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904279 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904522 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904547 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:18:54.904550 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904547 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sv7k2\"" Oct 31 03:18:54.904772 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.904557 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:18:54.907785 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-device-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.907828 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.907828 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.907828 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-conf-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.907950 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l97\" (UniqueName: \"kubernetes.io/projected/20d028be-4c41-438b-86ac-2e2eadce25df-kube-api-access-p4l97\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.907950 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907917 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-conf-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.907950 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907922 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-device-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.907950 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-ovn\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907962 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-ovn\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907947 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-host\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.907999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-tuned\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908007 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-host\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56e24d88-5120-43d1-9649-9245d1967788-iptables-alerter-script\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-var-lib-kubelet\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908128 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-registration-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-multus-daemon-config\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-registration-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cnibin\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrk8\" (UniqueName: \"kubernetes.io/projected/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-kube-api-access-xzrk8\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908203 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-var-lib-kubelet\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-run\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908262 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cnibin\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908276 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-run\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-kubelet-config\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-sys-fs\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-kubelet-config\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-system-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-env-overrides\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908364 2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:18:54.908439 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908372 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-system-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvhk\" (UniqueName: \"kubernetes.io/projected/0c92072e-6213-4831-b222-039305a4d122-kube-api-access-snvhk\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-modprobe-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-k8s-cni-cncf-io\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908457 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-multus-certs\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-etc-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-log-socket\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-dbus\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-sys-fs\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-sys\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908634 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56e24d88-5120-43d1-9649-9245d1967788-iptables-alerter-script\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908657 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-bin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908662 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-k8s-cni-cncf-io\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-bin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-multus-daemon-config\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908681 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-etc-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909049 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908695 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-multus-certs\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908705 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-modprobe-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-multus\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908724 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-dbus\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-multus\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908726 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-log-socket\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-kubelet\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-sys\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908764 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-slash\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-env-overrides\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908788 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-kubelet\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908821 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-slash\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908837 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-conf\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908857 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rj4z\" (UniqueName: \"kubernetes.io/projected/c436b9f3-e0e4-4662-98bf-a140ef5c010c-kube-api-access-5rj4z\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-systemd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.909901 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.908903 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908914 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-var-lib-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-config\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.908954 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:55.408935546 +0000 UTC m=+3.122894833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908959 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-systemd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-conf\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c92072e-6213-4831-b222-039305a4d122-ovn-node-metrics-cert\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.908981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-var-lib-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae4c6a55-1343-49ab-b514-78564e0f1d75-agent-certs\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysconfig\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zjq\" (UniqueName: \"kubernetes.io/projected/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-kube-api-access-f7zjq\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909120 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysconfig\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8hv\" (UniqueName: \"kubernetes.io/projected/9de23b18-25f8-43c3-9463-152099741b6b-kube-api-access-zd8hv\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-system-cni-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ttk\" (UniqueName: \"kubernetes.io/projected/de3f9807-0638-4b57-b00a-bb9bc112505b-kube-api-access-t7ttk\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436b9f3-e0e4-4662-98bf-a140ef5c010c-tmp-dir\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.910679 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-kubelet\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8th4\" (UniqueName: \"kubernetes.io/projected/56e24d88-5120-43d1-9649-9245d1967788-kube-api-access-f8th4\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909347 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-kubelet\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-system-cni-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-hostroot\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909449 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909464 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-config\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-etc-kubernetes\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56e24d88-5120-43d1-9649-9245d1967788-host-slash\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909563 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-etc-kubernetes\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909548 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-hostroot\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56e24d88-5120-43d1-9649-9245d1967788-host-slash\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9de23b18-25f8-43c3-9463-152099741b6b-serviceca\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-systemd\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c436b9f3-e0e4-4662-98bf-a140ef5c010c-hosts-file\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.911407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de23b18-25f8-43c3-9463-152099741b6b-host\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-systemd-units\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909754 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-systemd\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de23b18-25f8-43c3-9463-152099741b6b-host\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909836 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c436b9f3-e0e4-4662-98bf-a140ef5c010c-hosts-file\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-tmp\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909886 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-systemd-units\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-cnibin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-socket-dir-parent\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-netns\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909968 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-bin\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909968 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.909993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-socket-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910027 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9de23b18-25f8-43c3-9463-152099741b6b-serviceca\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.912058 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfg98\" (UniqueName: \"kubernetes.io/projected/449c2d15-e376-4ba5-a668-9c37ea713e3f-kube-api-access-gfg98\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910071 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-netd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-script-lib\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910129 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-netns\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910187 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-socket-dir-parent\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c6a55-1343-49ab-b514-78564e0f1d75-konnectivity-ca\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910205 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436b9f3-e0e4-4662-98bf-a140ef5c010c-tmp-dir\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-kubernetes\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910226 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-socket-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-bin\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910265 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-netd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-cni-binary-copy\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910305 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-netns\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.910326 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:54.912546 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-node-log\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.910374 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret podName:d66b1354-90f5-43ec-b395-cabd3dc171ac nodeName:}" failed. No retries permitted until 2026-10-31 03:18:55.41035685 +0000 UTC m=+3.124316153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret") pod "global-pull-secret-syncer-jvb8m" (UID: "d66b1354-90f5-43ec-b395-cabd3dc171ac") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910386 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-node-log\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-lib-modules\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-netns\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910480 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-cnibin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-lib-modules\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.910604 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-kubernetes\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.911275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-cni-binary-copy\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.911557 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-script-lib\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.911579 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c6a55-1343-49ab-b514-78564e0f1d75-konnectivity-ca\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.911962 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-tuned\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.911999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.912036 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.912145 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae4c6a55-1343-49ab-b514-78564e0f1d75-agent-certs\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.912268 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-tmp\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.912952 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.912396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c92072e-6213-4831-b222-039305a4d122-ovn-node-metrics-cert\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.962993 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.962955 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:54.962993 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.962993 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:54.963299 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.963007 2569 projected.go:196] Error preparing data for projected volume kube-api-access-t4nfr for pod openshift-network-diagnostics/network-check-target-h4tqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:54.963299 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:54.963112 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr podName:dcebf912-082c-462b-80d4-aba939d35610 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:55.463061115 +0000 UTC m=+3.177020415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t4nfr" (UniqueName: "kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr") pod "network-check-target-h4tqk" (UID: "dcebf912-082c-462b-80d4-aba939d35610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:54.966131 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966020 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rj4z\" (UniqueName: \"kubernetes.io/projected/c436b9f3-e0e4-4662-98bf-a140ef5c010c-kube-api-access-5rj4z\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:54.966131 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966032 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8hv\" (UniqueName: \"kubernetes.io/projected/9de23b18-25f8-43c3-9463-152099741b6b-kube-api-access-zd8hv\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:54.966131 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l97\" (UniqueName: \"kubernetes.io/projected/20d028be-4c41-438b-86ac-2e2eadce25df-kube-api-access-p4l97\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:18:54.966354 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966175 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zjq\" (UniqueName: \"kubernetes.io/projected/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-kube-api-access-f7zjq\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:54.966354 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966264 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8th4\" (UniqueName: \"kubernetes.io/projected/56e24d88-5120-43d1-9649-9245d1967788-kube-api-access-f8th4\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:54.966354 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966270 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrk8\" (UniqueName: \"kubernetes.io/projected/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-kube-api-access-xzrk8\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:54.966354 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966307 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvhk\" (UniqueName: \"kubernetes.io/projected/0c92072e-6213-4831-b222-039305a4d122-kube-api-access-snvhk\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:54.967000 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.966983 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ttk\" (UniqueName: \"kubernetes.io/projected/de3f9807-0638-4b57-b00a-bb9bc112505b-kube-api-access-t7ttk\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:54.973306 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:54.973283 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfg98\" (UniqueName: \"kubernetes.io/projected/449c2d15-e376-4ba5-a668-9c37ea713e3f-kube-api-access-gfg98\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:55.072509 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.072481 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:18:55.078664 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.078646 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vl82x" Oct 31 03:18:55.082365 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.082323 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0186cf5_c4bb_461d_90f7_d7c3f9bb9754.slice/crio-db70463432420bfe436405dc0293c4a5883d40a232824732b4037a14a739a619 WatchSource:0}: Error finding container db70463432420bfe436405dc0293c4a5883d40a232824732b4037a14a739a619: Status 404 returned error can't find the container with id db70463432420bfe436405dc0293c4a5883d40a232824732b4037a14a739a619 Oct 31 03:18:55.087055 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.087032 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:18:55.087334 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.087312 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc436b9f3_e0e4_4662_98bf_a140ef5c010c.slice/crio-8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9 WatchSource:0}: Error finding container 8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9: Status 404 returned error can't find the container with id 8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9 Oct 31 03:18:55.095599 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.095573 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de23b18_25f8_43c3_9463_152099741b6b.slice/crio-8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4 WatchSource:0}: Error finding container 8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4: Status 404 returned error can't find the container with id 8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4 Oct 31 03:18:55.111910 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.111876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:18:55.115782 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.115761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7w79" Oct 31 03:18:55.120196 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.120163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:18:55.120898 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.120873 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:18:55.125403 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.125228 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:18:55.129016 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.128993 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:18:55.129404 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.129376 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d028be_4c41_438b_86ac_2e2eadce25df.slice/crio-5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0 WatchSource:0}: Error finding container 5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0: Status 404 returned error can't find the container with id 5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0 Oct 31 03:18:55.137955 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.137924 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4c6a55_1343_49ab_b514_78564e0f1d75.slice/crio-614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842 WatchSource:0}: Error finding container 614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842: Status 404 returned error can't find the container with id 614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842 Oct 31 03:18:55.139124 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.139075 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e24d88_5120_43d1_9649_9245d1967788.slice/crio-1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6 WatchSource:0}: Error finding container 1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6: Status 404 returned error can't find the container with id 1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6 Oct 31 03:18:55.142376 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:18:55.142352 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e427ca_6b3a_4cf5_aff3_f0f7cdcf77b1.slice/crio-a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5 WatchSource:0}: Error finding container a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5: Status 404 returned error can't find the container with id a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5 Oct 31 03:18:55.204330 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.204254 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:18:55.415035 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.414996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:55.415240 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.415103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:55.415304 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.415244 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:55.415360 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.415305 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:56.415286295 +0000 UTC m=+4.129245582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:55.415414 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.415374 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:55.415414 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.415405 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret podName:d66b1354-90f5-43ec-b395-cabd3dc171ac nodeName:}" failed. No retries permitted until 2026-10-31 03:18:56.415394526 +0000 UTC m=+4.129353812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret") pod "global-pull-secret-syncer-jvb8m" (UID: "d66b1354-90f5-43ec-b395-cabd3dc171ac") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:55.516417 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.516324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:55.516580 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.516520 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:55.516580 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.516543 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:55.516580 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.516556 2569 projected.go:196] Error preparing data for projected volume kube-api-access-t4nfr for pod openshift-network-diagnostics/network-check-target-h4tqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:55.516701 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:55.516619 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr podName:dcebf912-082c-462b-80d4-aba939d35610 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:56.516601911 +0000 UTC m=+4.230561205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4nfr" (UniqueName: "kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr") pod "network-check-target-h4tqk" (UID: "dcebf912-082c-462b-80d4-aba939d35610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:55.779762 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.779610 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerStarted","Data":"a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5"} Oct 31 03:18:55.781232 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.781122 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnthq" event={"ID":"56e24d88-5120-43d1-9649-9245d1967788","Type":"ContainerStarted","Data":"1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6"} Oct 31 03:18:55.782364 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.782338 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kms5j" event={"ID":"ae4c6a55-1343-49ab-b514-78564e0f1d75","Type":"ContainerStarted","Data":"614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842"} Oct 31 03:18:55.783598 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.783576 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7w79" event={"ID":"20d028be-4c41-438b-86ac-2e2eadce25df","Type":"ContainerStarted","Data":"5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0"} Oct 31 03:18:55.784925 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.784896 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bljwr" event={"ID":"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754","Type":"ContainerStarted","Data":"db70463432420bfe436405dc0293c4a5883d40a232824732b4037a14a739a619"} Oct 31 03:18:55.785998 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.785974 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" event={"ID":"449c2d15-e376-4ba5-a668-9c37ea713e3f","Type":"ContainerStarted","Data":"9d2279f348c7640fe269f6f8d00eb915d8a22baa73518e0b28eb24cfdf32835b"} Oct 31 03:18:55.787253 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.787232 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"3c864de68ffb1d7aa1ecc8fef481a71dcef884c627cf8f7e302a12c0261541d3"} Oct 31 03:18:55.788543 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.788496 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftcg9" event={"ID":"9de23b18-25f8-43c3-9463-152099741b6b","Type":"ContainerStarted","Data":"8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4"} Oct 31 03:18:55.789685 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.789658 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vl82x" event={"ID":"c436b9f3-e0e4-4662-98bf-a140ef5c010c","Type":"ContainerStarted","Data":"8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9"} Oct 31 03:18:55.857630 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.857585 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:54 +0000 UTC" deadline="2028-06-28 02:51:57.428386576 +0000 UTC" Oct 31 03:18:55.857630 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:55.857624 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14543h33m1.570767107s" Oct 31 03:18:56.423446 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.423404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:56.423608 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.423480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:56.423665 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.423636 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:56.423712 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.423696 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret podName:d66b1354-90f5-43ec-b395-cabd3dc171ac nodeName:}" failed. No retries permitted until 2026-10-31 03:18:58.423679474 +0000 UTC m=+6.137638770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret") pod "global-pull-secret-syncer-jvb8m" (UID: "d66b1354-90f5-43ec-b395-cabd3dc171ac") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:56.423789 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.423757 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:56.423836 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.423793 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:58.423782016 +0000 UTC m=+6.137741304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:56.524484 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.524446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:56.524635 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.524618 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:56.524700 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.524642 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:56.524700 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.524654 2569 projected.go:196] Error preparing data for projected volume kube-api-access-t4nfr for pod openshift-network-diagnostics/network-check-target-h4tqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:56.524798 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.524708 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr podName:dcebf912-082c-462b-80d4-aba939d35610 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:58.524691776 +0000 UTC m=+6.238651061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4nfr" (UniqueName: "kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr") pod "network-check-target-h4tqk" (UID: "dcebf912-082c-462b-80d4-aba939d35610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:56.773658 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.773574 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:56.773815 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.773719 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:18:56.773877 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.773827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:56.773927 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.773908 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:18:56.773988 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.773974 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:56.774241 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:56.774214 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:18:56.858024 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.857974 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:54 +0000 UTC" deadline="2028-08-06 15:24:01.243423142 +0000 UTC" Oct 31 03:18:56.858024 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:56.858021 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15492h5m4.385413021s" Oct 31 03:18:57.804284 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:57.804247 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"abb3780775d238ba0d694f1e9cc7fb2a","Type":"ContainerStarted","Data":"04bebd4b09a044bf043a87bdabc043ecd869c15fa5cb6249ea964ca099b01eb6"} Oct 31 03:18:57.816917 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:57.816877 2569 generic.go:356] "Generic (PLEG): container finished" podID="0cd30d86732a0a5d7b8493f1d7958e8b" containerID="bc96389109a4d5899d3029fd619e07c1037b66a695467c1daefb29c5b3ac9c2a" exitCode=0 Oct 31 03:18:57.817154 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:57.816937 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"0cd30d86732a0a5d7b8493f1d7958e8b","Type":"ContainerDied","Data":"bc96389109a4d5899d3029fd619e07c1037b66a695467c1daefb29c5b3ac9c2a"} Oct 31 03:18:58.036823 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.036765 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" podStartSLOduration=4.036746266 podStartE2EDuration="4.036746266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:57.920728188 +0000 UTC m=+5.634687495" watchObservedRunningTime="2026-10-31 03:18:58.036746266 +0000 UTC m=+5.750705574" Oct 31 03:18:58.448703 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.448666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:58.448866 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.448726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:58.448866 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.448850 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:58.448989 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.448896 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret podName:d66b1354-90f5-43ec-b395-cabd3dc171ac nodeName:}" failed. No retries permitted until 2026-10-31 03:19:02.448884441 +0000 UTC m=+10.162843728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret") pod "global-pull-secret-syncer-jvb8m" (UID: "d66b1354-90f5-43ec-b395-cabd3dc171ac") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:18:58.448989 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.448947 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:58.448989 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.448968 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:02.448961407 +0000 UTC m=+10.162920690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:18:58.549628 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.549590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:58.549767 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.549757 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:18:58.549823 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.549775 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:18:58.549823 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.549789 2569 projected.go:196] Error preparing data for projected volume kube-api-access-t4nfr for pod openshift-network-diagnostics/network-check-target-h4tqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:58.549914 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.549853 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr podName:dcebf912-082c-462b-80d4-aba939d35610 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:02.549833804 +0000 UTC m=+10.263793106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4nfr" (UniqueName: "kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr") pod "network-check-target-h4tqk" (UID: "dcebf912-082c-462b-80d4-aba939d35610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:18:58.773504 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.773468 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:18:58.773690 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.773617 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:18:58.773976 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.773954 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:18:58.774138 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.774064 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:18:58.774265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.774161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:18:58.774265 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:18:58.774252 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:18:58.822640 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.822591 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="21cd0ce10c70a6eeb8662a6ba4b738fd83aa8b0de56968d3eb76b7665131a3cf" exitCode=0 Oct 31 03:18:58.822818 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.822685 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerDied","Data":"21cd0ce10c70a6eeb8662a6ba4b738fd83aa8b0de56968d3eb76b7665131a3cf"} Oct 31 03:18:58.829969 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:58.829859 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"0cd30d86732a0a5d7b8493f1d7958e8b","Type":"ContainerStarted","Data":"9a08c1f524325de640dddc097374ec5120776bf31b2d543c2134d28a9e60a1f9"} Oct 31 03:18:59.844136 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:59.842231 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kms5j" event={"ID":"ae4c6a55-1343-49ab-b514-78564e0f1d75","Type":"ContainerStarted","Data":"de47fac93304fe5734092bdd3ea188d23d3e04a4b9d7277cb11ddaaad47e504d"} Oct 31 03:18:59.958556 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:59.958496 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" podStartSLOduration=6.958474148 podStartE2EDuration="6.958474148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.064051614 +0000 UTC m=+6.778010920" watchObservedRunningTime="2026-10-31 03:18:59.958474148 +0000 UTC m=+7.672433450" Oct 31 03:18:59.959457 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:18:59.959404 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kms5j" podStartSLOduration=3.700860418 podStartE2EDuration="7.959386754s" totalImagesPullingTime="4.258526336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.95838806 +0000 UTC m=+7.672347366" watchObservedRunningTime="2026-10-31 03:18:59.959386754 +0000 UTC m=+7.673346062" Oct 31 03:19:00.612648 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.612484 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:00.613054 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.613034 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:00.772268 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.772240 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:00.772445 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:00.772359 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:00.772445 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.772426 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:00.772618 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:00.772508 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:00.772618 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.772573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:00.772717 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:00.772642 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:00.844748 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.844713 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftcg9" event={"ID":"9de23b18-25f8-43c3-9463-152099741b6b","Type":"ContainerStarted","Data":"27350c31613083f2117660319fa08e9b5c339929ff9c54affae6d88c89448090"} Oct 31 03:19:00.845331 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.844771 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:00.845331 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:00.845285 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:01.077296 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:01.077231 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ftcg9" podStartSLOduration=3.6101280129999997 podStartE2EDuration="9.07721029s" totalImagesPullingTime="5.467082277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:00.961071434 +0000 UTC m=+8.675030741" watchObservedRunningTime="2026-10-31 03:19:01.07721029 +0000 UTC m=+8.791169596" Oct 31 03:19:01.850467 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:01.850346 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" event={"ID":"449c2d15-e376-4ba5-a668-9c37ea713e3f","Type":"ContainerStarted","Data":"8f374afe845484232fcf8218003528b89a6f09827b66acd1f9a0fb7291c8b399"} Oct 31 03:19:02.483241 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:02.483200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:02.483430 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:02.483297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:02.483512 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.483474 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:19:02.483567 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.483538 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:10.483519879 +0000 UTC m=+18.197479168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:19:02.484153 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.483955 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:19:02.484153 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.484011 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret podName:d66b1354-90f5-43ec-b395-cabd3dc171ac nodeName:}" failed. No retries permitted until 2026-10-31 03:19:10.48399444 +0000 UTC m=+18.197953744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret") pod "global-pull-secret-syncer-jvb8m" (UID: "d66b1354-90f5-43ec-b395-cabd3dc171ac") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:19:02.584121 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:02.583941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:02.584121 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.584113 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:19:02.584121 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.584135 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:19:02.584359 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.584147 2569 projected.go:196] Error preparing data for projected volume kube-api-access-t4nfr for pod openshift-network-diagnostics/network-check-target-h4tqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:19:02.584359 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.584202 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr podName:dcebf912-082c-462b-80d4-aba939d35610 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:10.584185378 +0000 UTC m=+18.298144667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4nfr" (UniqueName: "kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr") pod "network-check-target-h4tqk" (UID: "dcebf912-082c-462b-80d4-aba939d35610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:19:02.775523 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:02.775443 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:02.775695 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.775585 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:02.776052 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:02.776028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:02.776183 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.776159 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:02.776357 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:02.776337 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:02.776464 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:02.776444 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:03.173685 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.173632 2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:19:03.766599 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.766473 2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:19:03.1736569Z","UUID":"3521619c-b5d0-4a57-8639-294c63dd0892","Handler":null,"Name":"","Endpoint":""} Oct 31 03:19:03.769600 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.768987 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:19:03.769600 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.769021 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:19:03.858509 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.858369 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" event={"ID":"449c2d15-e376-4ba5-a668-9c37ea713e3f","Type":"ContainerStarted","Data":"2c3efe89abdaafd6f91f905987b611311ec90fe2cf53b5404bac39f8cc065f33"} Oct 31 03:19:03.863473 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.863445 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7w79" event={"ID":"20d028be-4c41-438b-86ac-2e2eadce25df","Type":"ContainerStarted","Data":"1f5066bb9759748c4876d306589193b5126f81ed1a32a1a907bc52aaf1da5392"} Oct 31 03:19:03.983060 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:03.982964 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r7w79" podStartSLOduration=3.864947216 podStartE2EDuration="11.982947439s" totalImagesPullingTime="8.118000223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:03.982703544 +0000 UTC m=+11.696662871" watchObservedRunningTime="2026-10-31 03:19:03.982947439 +0000 UTC m=+11.696906742" Oct 31 03:19:04.773633 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.773392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:04.774243 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:04.773720 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:04.774243 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.773796 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:04.774243 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:04.773859 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:04.774243 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.773916 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:04.774243 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:04.773973 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:04.869347 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.868631 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="5f05aa69816841d897034c026218a249cf62c8f09e755817292a1f2de37f3e4a" exitCode=0 Oct 31 03:19:04.869347 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.868751 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerDied","Data":"5f05aa69816841d897034c026218a249cf62c8f09e755817292a1f2de37f3e4a"} Oct 31 03:19:04.872000 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.871461 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" event={"ID":"449c2d15-e376-4ba5-a668-9c37ea713e3f","Type":"ContainerStarted","Data":"80e70a14d3e8eb6dfce391fb96b9c6662255df4cb47ce82e6f522ccb597d4a24"} Oct 31 03:19:04.874312 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:04.874273 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vl82x" event={"ID":"c436b9f3-e0e4-4662-98bf-a140ef5c010c","Type":"ContainerStarted","Data":"38cb95cb5e2ceb250dd8321dadfc93700681008111cadc415354d4896dd9bd40"} Oct 31 03:19:05.155842 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:05.155724 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vl82x" podStartSLOduration=4.543969667 podStartE2EDuration="13.155704209s" totalImagesPullingTime="8.611734542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:05.155313368 +0000 UTC m=+12.869272675" watchObservedRunningTime="2026-10-31 03:19:05.155704209 +0000 UTC m=+12.869663517" Oct 31 03:19:05.275444 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:05.275375 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" podStartSLOduration=4.263606699 podStartE2EDuration="13.275352276s" totalImagesPullingTime="9.011745577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:05.27446541 +0000 UTC m=+12.988424720" watchObservedRunningTime="2026-10-31 03:19:05.275352276 +0000 UTC m=+12.989311584" Oct 31 03:19:05.877769 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:05.877736 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnthq" event={"ID":"56e24d88-5120-43d1-9649-9245d1967788","Type":"ContainerStarted","Data":"388c429e836d6736386a24016157bae66736c58327993ca8cacc8e60c687da09"} Oct 31 03:19:05.879513 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:05.879475 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bljwr" event={"ID":"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754","Type":"ContainerStarted","Data":"de0448e402cac4a229526bb6085bcd7fb21f3e2ce9d1364421b7b816c0fa9dec"} Oct 31 03:19:05.992653 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:05.992593 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cnthq" podStartSLOduration=5.689378784 podStartE2EDuration="13.992577689s" totalImagesPullingTime="8.303198905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:05.992268523 +0000 UTC m=+13.706227833" watchObservedRunningTime="2026-10-31 03:19:05.992577689 +0000 UTC m=+13.706536995" Oct 31 03:19:06.771138 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:06.771110 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:06.771315 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:06.771107 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:06.771386 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:06.771353 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:06.771386 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:06.771122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:06.771485 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:06.771225 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:06.771531 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:06.771477 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:06.884005 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:06.883968 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="59fd3eb0a1423e073b839c8039199ca8574a405dc0a0eea5483ae64f380ff9e1" exitCode=0 Oct 31 03:19:06.884614 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:06.884053 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerDied","Data":"59fd3eb0a1423e073b839c8039199ca8574a405dc0a0eea5483ae64f380ff9e1"} Oct 31 03:19:07.156446 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:07.156385 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bljwr" podStartSLOduration=4.679359509 podStartE2EDuration="15.156370958s" totalImagesPullingTime="10.477011449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:06.110728078 +0000 UTC m=+13.824687383" watchObservedRunningTime="2026-10-31 03:19:07.156370958 +0000 UTC m=+14.870330263" Oct 31 03:19:08.771365 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:08.771325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:08.771365 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:08.771373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:08.772075 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:08.771345 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:08.772075 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:08.771478 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:08.772075 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:08.771528 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:08.772075 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:08.771625 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:08.890663 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:08.890626 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="65bcbbe57d26ae259a709beca920a23a905d58beb3a0bbf9bf806cd5d3299902" exitCode=0 Oct 31 03:19:08.890860 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:08.890677 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerDied","Data":"65bcbbe57d26ae259a709beca920a23a905d58beb3a0bbf9bf806cd5d3299902"} Oct 31 03:19:09.899917 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:09.899674 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"83f235ee1d5c510564b6b8cd9396502730a989791d1d619895d783cba8b971ca"} Oct 31 03:19:09.900470 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:09.899962 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"a8a2aba721c6d18bc080975e292c9967ee89d766ca953b61e8d36adb7eec8006"} Oct 31 03:19:09.900470 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:09.899999 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"d25b08a0b29ef2f9781dcd0c8465c2aeddeb69e89f7042e77f896eb5dad0e706"} Oct 31 03:19:09.900470 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:09.900034 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"0ee9e7ec80c5291dcca0233c1d7b1db26936000ae96f8ce428ad5f932368835a"} Oct 31 03:19:09.900470 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:09.900065 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"0b9faf0cbdd94d06aec96b8cbe65ec3258877178b8259d4a07a0338e065c3d05"} Oct 31 03:19:09.900470 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:09.900119 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"ba5ac590f6ff488209f821e79ac01ba55a6307e8847a74ebd27b04f022e31bbe"} Oct 31 03:19:10.549556 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:10.549367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:10.549726 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:10.549603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:10.549726 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.549486 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:19:10.549820 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.549754 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret podName:d66b1354-90f5-43ec-b395-cabd3dc171ac nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.549742031 +0000 UTC m=+34.263701313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret") pod "global-pull-secret-syncer-jvb8m" (UID: "d66b1354-90f5-43ec-b395-cabd3dc171ac") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:19:10.550066 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.549706 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:19:10.550224 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.550121 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.550103141 +0000 UTC m=+34.264062444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:19:10.649971 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:10.649922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:10.650135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.650082 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:19:10.650135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.650122 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:19:10.650135 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.650134 2569 projected.go:196] Error preparing data for projected volume kube-api-access-t4nfr for pod openshift-network-diagnostics/network-check-target-h4tqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:19:10.650278 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.650180 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr podName:dcebf912-082c-462b-80d4-aba939d35610 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.650168086 +0000 UTC m=+34.364127381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4nfr" (UniqueName: "kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr") pod "network-check-target-h4tqk" (UID: "dcebf912-082c-462b-80d4-aba939d35610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:19:10.771335 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:10.771300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:10.771511 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:10.771365 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:10.771511 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:10.771442 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:10.771511 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.771453 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:10.771674 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.771548 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:10.771674 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:10.771613 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:11.908183 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:11.908152 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"0325100ef9c809da7a70a13d3121466c12f5a43a4e2d72fb9c882846fd115ad7"} Oct 31 03:19:11.911142 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:11.911112 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="31ac82a13e8d749fc64531c690e7057bf617d1db9f08c88f471457b945001bce" exitCode=0 Oct 31 03:19:11.911278 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:11.911123 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerDied","Data":"31ac82a13e8d749fc64531c690e7057bf617d1db9f08c88f471457b945001bce"} Oct 31 03:19:12.771070 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:12.771043 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:12.771270 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:12.771164 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:12.771270 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:12.771232 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:12.771406 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:12.771340 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:12.771406 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:12.771385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:12.771499 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:12.771461 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:12.916826 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:12.916784 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="7a371b0449780c7babd4385d7253069c42b66e5244603372968745ae463a0c0d" exitCode=0 Oct 31 03:19:12.917338 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:12.916866 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerDied","Data":"7a371b0449780c7babd4385d7253069c42b66e5244603372968745ae463a0c0d"} Oct 31 03:19:13.923139 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:13.923107 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" event={"ID":"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1","Type":"ContainerStarted","Data":"e3427329f1ef264a26fc1317e10498b7e23593d840213b472204702959921e75"} Oct 31 03:19:14.151669 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:14.151406 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fpp2c" podStartSLOduration=10.074691875 podStartE2EDuration="22.151386608s" totalImagesPullingTime="12.076694733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:14.151011516 +0000 UTC m=+21.864970821" watchObservedRunningTime="2026-10-31 03:19:14.151386608 +0000 UTC m=+21.865345916" Oct 31 03:19:14.771252 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:14.771211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:14.771252 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:14.771250 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:14.771475 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:14.771353 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:14.771475 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:14.771422 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:14.771475 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:14.771440 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:14.771597 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:14.771517 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:14.929522 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:14.929477 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" event={"ID":"0c92072e-6213-4831-b222-039305a4d122","Type":"ContainerStarted","Data":"fda8a234ecbae6a3dbb977fcac788914be86bac2bbf31553bdf7c8c665fd4768"} Oct 31 03:19:15.170672 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:15.170612 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" podStartSLOduration=9.02475441 podStartE2EDuration="23.170593284s" totalImagesPullingTime="14.145838874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:15.119910281 +0000 UTC m=+22.833869587" watchObservedRunningTime="2026-10-31 03:19:15.170593284 +0000 UTC m=+22.884552591" Oct 31 03:19:15.931866 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:15.931827 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:15.932335 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:15.932309 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:15.932476 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:15.932427 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:15.949694 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:15.949670 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:15.949840 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:15.949787 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:16.770494 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.770402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:16.770494 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.770411 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:16.770759 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:16.770578 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:16.770872 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.770858 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:16.770950 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:16.770935 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:16.771050 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:16.771011 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:16.773740 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.773716 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jvb8m"] Oct 31 03:19:16.774458 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.774435 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x75qg"] Oct 31 03:19:16.775216 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.775189 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4tqk"] Oct 31 03:19:16.933695 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.933671 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:16.933695 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.933694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:16.934190 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.933750 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:16.934190 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:16.933671 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:16.934190 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:16.933780 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:16.934190 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:16.934046 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:16.934190 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:16.934136 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:17.320474 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:17.320431 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.560080 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:17.560040 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.761975 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:17.761938 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.936074 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:17.936052 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:18.162055 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.162023 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:18.181566 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.181514 2569 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" podUID="0c92072e-6213-4831-b222-039305a4d122" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 03:19:18.191055 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.191022 2569 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" podUID="0c92072e-6213-4831-b222-039305a4d122" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 03:19:18.469697 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.469662 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:18.620512 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.620482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:18.770910 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.770822 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:18.770910 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.770830 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:18.771165 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:18.770935 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:18.771165 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.770944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:18.771165 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:18.771010 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:18.771165 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:18.771112 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:18.949220 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:18.949182 2569 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" podUID="0c92072e-6213-4831-b222-039305a4d122" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 03:19:19.420141 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:19.420110 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:19.622719 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:19.622688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:20.598716 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:20.598687 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:20.753862 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:20.753813 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:20.770489 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:20.770466 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:20.770609 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:20.770467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:20.770609 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:20.770581 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x75qg" podUID="de3f9807-0638-4b57-b00a-bb9bc112505b" Oct 31 03:19:20.770699 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:20.770478 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:20.770730 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:20.770630 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvb8m" podUID="d66b1354-90f5-43ec-b395-cabd3dc171ac" Oct 31 03:19:20.770730 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:20.770710 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tqk" podUID="dcebf912-082c-462b-80d4-aba939d35610" Oct 31 03:19:22.113230 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.113020 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeReady" Oct 31 03:19:22.113565 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.113339 2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:19:22.229393 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.229306 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x5fwg"] Oct 31 03:19:22.237728 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.237693 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6bbvw"] Oct 31 03:19:22.237874 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.237802 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.247884 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.247865 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.298105 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.298043 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:19:22.298262 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.298138 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-n6jvn\"" Oct 31 03:19:22.298262 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.298180 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:19:22.298660 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.298646 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:19:22.307947 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.307921 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5fwg"] Oct 31 03:19:22.308518 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.308486 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bbvw"] Oct 31 03:19:22.335157 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.335130 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shfs\" (UniqueName: \"kubernetes.io/projected/8bf3c83e-4db7-4844-8aca-c66569729d3b-kube-api-access-5shfs\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.335308 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.335163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-config-volume\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.335308 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.335190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-tmp-dir\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.335308 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.335264 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.335407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.335341 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bf3c83e-4db7-4844-8aca-c66569729d3b-cert\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.335407 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.335363 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84lg\" (UniqueName: \"kubernetes.io/projected/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-kube-api-access-c84lg\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.392384 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.392348 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:19:22.392561 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.392417 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:19:22.392561 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.392496 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btzr7\"" Oct 31 03:19:22.436670 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.436637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shfs\" (UniqueName: \"kubernetes.io/projected/8bf3c83e-4db7-4844-8aca-c66569729d3b-kube-api-access-5shfs\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.436825 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.436677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-config-volume\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.436825 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.436716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-tmp-dir\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.436825 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.436738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.436825 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.436766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bf3c83e-4db7-4844-8aca-c66569729d3b-cert\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.436825 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.436786 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84lg\" (UniqueName: \"kubernetes.io/projected/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-kube-api-access-c84lg\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.437120 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.436892 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:22.437120 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.436965 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:22.936943057 +0000 UTC m=+30.650902354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:22.437229 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.437188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-tmp-dir\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.437406 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.437389 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-config-volume\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.441348 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.441325 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bf3c83e-4db7-4844-8aca-c66569729d3b-cert\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.501332 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.501258 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shfs\" (UniqueName: \"kubernetes.io/projected/8bf3c83e-4db7-4844-8aca-c66569729d3b-kube-api-access-5shfs\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.501332 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.501287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84lg\" (UniqueName: \"kubernetes.io/projected/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-kube-api-access-c84lg\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.560176 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.560146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:22.633287 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.633251 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qs77w"] Oct 31 03:19:22.662352 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.662324 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.699079 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.699050 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qs77w"] Oct 31 03:19:22.738993 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.738958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9890fd99-71b5-45ab-a4f0-38293eff7802-data-volume\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.739163 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.739024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg9l5\" (UniqueName: \"kubernetes.io/projected/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-api-access-rg9l5\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.739163 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.739115 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9890fd99-71b5-45ab-a4f0-38293eff7802-crio-socket\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.739163 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.739143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.739163 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.739159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.770349 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.770274 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:22.770349 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.770293 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:22.770349 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.770308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:22.797972 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.797946 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:19:22.798110 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.798016 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:19:22.798187 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.798174 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:19:22.804661 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.804647 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dbgqm\"" Oct 31 03:19:22.804661 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.804657 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:19:22.809771 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.809749 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:22.822972 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.822953 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9vj2z-6h7j2"] Oct 31 03:19:22.825265 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.825249 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:19:22.825345 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.825253 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:19:22.829302 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.829277 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:22.838104 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.837491 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v5ddh\"" Oct 31 03:19:22.838104 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.837525 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkxjs\"" Oct 31 03:19:22.838104 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.837530 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:19:22.838104 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.837528 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:19:22.838104 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.837570 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vxflf\"" Oct 31 03:19:22.839558 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.839534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9890fd99-71b5-45ab-a4f0-38293eff7802-crio-socket\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.839647 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.839567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.839647 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.839591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.839647 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.839643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9890fd99-71b5-45ab-a4f0-38293eff7802-data-volume\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.839779 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.839670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9l5\" (UniqueName: \"kubernetes.io/projected/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-api-access-rg9l5\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.839779 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.839671 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:22.839779 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.839737 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:22.839779 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.839760 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:23.339742529 +0000 UTC m=+31.053701815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : configmap "kube-rbac-proxy" not found Oct 31 03:19:22.839779 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.839770 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9890fd99-71b5-45ab-a4f0-38293eff7802-crio-socket\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.839994 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.839791 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:23.339775802 +0000 UTC m=+31.053735105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:22.840048 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.840028 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9890fd99-71b5-45ab-a4f0-38293eff7802-data-volume\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.886046 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.886013 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9vj2z-6h7j2"] Oct 31 03:19:22.901497 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.901474 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9l5\" (UniqueName: \"kubernetes.io/projected/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-api-access-rg9l5\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:22.904498 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.904481 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jjf7q\"" Oct 31 03:19:22.925363 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.925326 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:19:22.940637 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.940612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:22.940732 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.940675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:22.940732 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.940695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:22.940799 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.940782 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:22.940830 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.940810 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:22.940858 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.940816 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:22.940889 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:22.940873 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:23.94085325 +0000 UTC m=+31.654812552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:22.965280 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.965240 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:22.976292 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:22.976269 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:19:23.015948 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.015914 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5fwg"] Oct 31 03:19:23.019561 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:19:23.019533 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf3c83e_4db7_4844_8aca_c66569729d3b.slice/crio-e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781 WatchSource:0}: Error finding container e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781: Status 404 returned error can't find the container with id e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781 Oct 31 03:19:23.041353 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.041314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.041483 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.041413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.041483 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.041468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.041599 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.041525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.041882 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.041861 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.042074 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.042054 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.042527 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.042511 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.070424 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.070393 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw"] Oct 31 03:19:23.103428 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.103396 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:23.105211 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.105184 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.128348 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.128321 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:23.131391 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.131366 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw"] Oct 31 03:19:23.146560 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.146536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:23.158363 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.158338 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 03:19:23.158563 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.158545 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:23.158634 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.158605 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b5nvx\"" Oct 31 03:19:23.243382 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.243357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxxj\" (UniqueName: \"kubernetes.io/projected/98836313-6e91-46d0-84b9-f4306b05a80d-kube-api-access-dtxxj\") pod \"migrator-66df5d496b-ffmpw\" (UID: \"98836313-6e91-46d0-84b9-f4306b05a80d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:23.288028 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.287913 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:23.344016 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.343982 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxxj\" (UniqueName: \"kubernetes.io/projected/98836313-6e91-46d0-84b9-f4306b05a80d-kube-api-access-dtxxj\") pod \"migrator-66df5d496b-ffmpw\" (UID: \"98836313-6e91-46d0-84b9-f4306b05a80d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:23.344211 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.344076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:23.344211 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.344131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:23.344291 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:23.344199 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:23.344291 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:23.344279 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.344258244 +0000 UTC m=+32.058217541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : configmap "kube-rbac-proxy" not found Oct 31 03:19:23.344291 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:23.344284 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:23.344452 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:23.344321 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.344311269 +0000 UTC m=+32.058270555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:23.406810 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.406777 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxxj\" (UniqueName: \"kubernetes.io/projected/98836313-6e91-46d0-84b9-f4306b05a80d-kube-api-access-dtxxj\") pod \"migrator-66df5d496b-ffmpw\" (UID: \"98836313-6e91-46d0-84b9-f4306b05a80d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:23.425896 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.425868 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:23.437037 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.436996 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:23.530450 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.530420 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9vj2z-6h7j2"] Oct 31 03:19:23.533277 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:19:23.533250 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef682d2d_6dfe_4b58_bf79_e7b2e268ff62.slice/crio-05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b WatchSource:0}: Error finding container 05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b: Status 404 returned error can't find the container with id 05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b Oct 31 03:19:23.794995 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.794963 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw"] Oct 31 03:19:23.798193 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:19:23.798167 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98836313_6e91_46d0_84b9_f4306b05a80d.slice/crio-f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee WatchSource:0}: Error finding container f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee: Status 404 returned error can't find the container with id f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee Oct 31 03:19:23.949249 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.949217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:23.949412 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:23.949380 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:23.949486 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:23.949463 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:25.949439901 +0000 UTC m=+33.663399201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:23.952747 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.952709 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" event={"ID":"98836313-6e91-46d0-84b9-f4306b05a80d","Type":"ContainerStarted","Data":"f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee"} Oct 31 03:19:23.954048 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.954021 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" event={"ID":"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62","Type":"ContainerStarted","Data":"05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b"} Oct 31 03:19:23.955218 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:23.955189 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5fwg" event={"ID":"8bf3c83e-4db7-4844-8aca-c66569729d3b","Type":"ContainerStarted","Data":"e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781"} Oct 31 03:19:24.003985 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:24.003945 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:24.205849 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:24.205802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:24.352566 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:24.352530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:24.352566 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:24.352567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:24.352812 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:24.352661 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:24.352812 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:24.352730 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:24.352812 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:24.352735 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.352715959 +0000 UTC m=+34.066675257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : configmap "kube-rbac-proxy" not found Oct 31 03:19:24.352812 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:24.352764 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:26.352754061 +0000 UTC m=+34.066713346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:24.598046 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:24.597833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:24.753118 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:24.753007 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:25.074213 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:25.074016 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:25.279510 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:25.279447 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:25.433600 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:25.433543 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:25.962603 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:25.962571 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" event={"ID":"98836313-6e91-46d0-84b9-f4306b05a80d","Type":"ContainerStarted","Data":"64c4fcae0980e27e152a1f1f93c52bd1379a2b107a571fa4c6bbcc6bc460fa7e"} Oct 31 03:19:25.967609 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:25.967584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:25.967797 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:25.967782 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:25.967867 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:25.967837 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:29.967822749 +0000 UTC m=+37.681782044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:26.079399 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.079345 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" podStartSLOduration=1.364376315 podStartE2EDuration="3.079329812s" totalImagesPullingTime="1.714953497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:26.078931802 +0000 UTC m=+33.792891107" watchObservedRunningTime="2026-10-31 03:19:26.079329812 +0000 UTC m=+33.793289114" Oct 31 03:19:26.370656 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.370621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:26.370656 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.370665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:26.371202 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:26.370751 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 03:19:26.371202 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:26.370792 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:26.371202 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:26.370816 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.37080105 +0000 UTC m=+38.084760335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : configmap "kube-rbac-proxy" not found Oct 31 03:19:26.371202 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:26.370845 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:30.370829799 +0000 UTC m=+38.084789088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:26.572932 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.572848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:26.572932 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.572897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:26.573162 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:26.572994 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 03:19:26.573162 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:26.573069 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs podName:de3f9807-0638-4b57-b00a-bb9bc112505b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:58.573053139 +0000 UTC m=+66.287012425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs") pod "network-metrics-daemon-x75qg" (UID: "de3f9807-0638-4b57-b00a-bb9bc112505b") : secret "metrics-daemon-secret" not found Oct 31 03:19:26.575658 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.575630 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:26.673406 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.673369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:26.676024 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.676001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:26.704032 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.704002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:26.710882 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.710857 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:26.967253 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.967218 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" event={"ID":"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62","Type":"ContainerStarted","Data":"b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4"} Oct 31 03:19:26.969053 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:26.969009 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5fwg" event={"ID":"8bf3c83e-4db7-4844-8aca-c66569729d3b","Type":"ContainerStarted","Data":"507df2e71f48d98d5227f2eae42f25d54d81f2772d74bb2f2ae9513e1dc633b3"} Oct 31 03:19:27.080963 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:27.080920 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4tqk"] Oct 31 03:19:27.084501 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:19:27.084473 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcebf912_082c_462b_80d4_aba939d35610.slice/crio-fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de WatchSource:0}: Error finding container fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de: Status 404 returned error can't find the container with id fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de Oct 31 03:19:27.131154 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:27.131049 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" podStartSLOduration=2.545305339 podStartE2EDuration="5.131025696s" totalImagesPullingTime="2.585720357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:27.083238298 +0000 UTC m=+34.797197601" watchObservedRunningTime="2026-10-31 03:19:27.131025696 +0000 UTC m=+34.844985002" Oct 31 03:19:27.132247 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:27.132228 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jvb8m"] Oct 31 03:19:27.134723 ip-10-0-143-24 kubenswrapper[2569]: W1031 03:19:27.134698 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b1354_90f5_43ec_b395_cabd3dc171ac.slice/crio-a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05 WatchSource:0}: Error finding container a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05: Status 404 returned error can't find the container with id a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05 Oct 31 03:19:27.203255 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:27.203196 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x5fwg" podStartSLOduration=2.099667278 podStartE2EDuration="5.203179245s" totalImagesPullingTime="3.103511967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:27.202411789 +0000 UTC m=+34.916371094" watchObservedRunningTime="2026-10-31 03:19:27.203179245 +0000 UTC m=+34.917138548" Oct 31 03:19:27.972935 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:27.972893 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jvb8m" event={"ID":"d66b1354-90f5-43ec-b395-cabd3dc171ac","Type":"ContainerStarted","Data":"a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05"} Oct 31 03:19:27.974318 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:27.974281 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4tqk" event={"ID":"dcebf912-082c-462b-80d4-aba939d35610","Type":"ContainerStarted","Data":"fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de"} Oct 31 03:19:28.427939 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:28.427884 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.083361 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:29.083323 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.243815 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:29.243727 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.553496 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:29.553407 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.703548 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:29.703510 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:30.004200 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.004159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:30.004383 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:30.004347 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:30.004438 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:30.004410 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:38.00439549 +0000 UTC m=+45.718354792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:30.196333 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.196265 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:30.256035 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.255960 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vl82x_c436b9f3-e0e4-4662-98bf-a140ef5c010c/dns-node-resolver/0.log" Oct 31 03:19:30.408039 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.408001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:30.408245 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.408045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:30.408317 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:30.408252 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:30.408364 ip-10-0-143-24 kubenswrapper[2569]: E1031 03:19:30.408326 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:38.408309338 +0000 UTC m=+46.122268620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:30.408584 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.408566 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:30.591417 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.591384 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:30.984052 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.984015 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4tqk" event={"ID":"dcebf912-082c-462b-80d4-aba939d35610","Type":"ContainerStarted","Data":"cb1274d9c11d6bd4141c0131b8bc687345c32afc03e81a57c9bae1130b50ff33"} Oct 31 03:19:30.984240 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.984141 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:30.992904 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:30.992868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.055767 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:31.055743 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftcg9_9de23b18-25f8-43c3-9463-152099741b6b/node-ca/0.log" Oct 31 03:19:31.100927 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:31.100856 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4tqk" podStartSLOduration=35.720777607 podStartE2EDuration="39.100838295s" totalImagesPullingTime="3.380060688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:31.100629302 +0000 UTC m=+38.814588609" watchObservedRunningTime="2026-10-31 03:19:31.100838295 +0000 UTC m=+38.814797601" Oct 31 03:19:31.388446 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:31.388412 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.676192 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:31.676108 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.828135 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:31.828076 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.988795 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:31.988749 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jvb8m" event={"ID":"d66b1354-90f5-43ec-b395-cabd3dc171ac","Type":"ContainerStarted","Data":"4b6f33d86d6d6bcac3123eb17dbba28b75f25e9b4931bca41a2cf61caf629a80"} Oct 31 03:19:32.055993 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:32.055966 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5fwg_8bf3c83e-4db7-4844-8aca-c66569729d3b/serve-healthcheck-canary/0.log" Oct 31 03:19:32.104917 ip-10-0-143-24 kubenswrapper[2569]: I1031 03:19:32.104842 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jvb8m" podStartSLOduration=35.38145054 podStartE2EDuration="40.104823276s" totalImagesPullingTime="4.723372736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:32.104723896 +0000 UTC m=+39.818683202" watchObservedRunningTime="2026-10-31 03:19:32.104823276 +0000 UTC m=+39.818782582" Oct 31 03:19:32.121077 ip-10-0-143-24 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 03:19:32.139696 ip-10-0-143-24 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 03:19:32.139886 ip-10-0-143-24 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 03:19:32.140117 ip-10-0-143-24 systemd[1]: kubelet.service: Consumed 2.611s CPU time. Oct 31 03:19:32.158617 ip-10-0-143-24 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:19:32.234248 ip-10-0-143-24 kubenswrapper[4488]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:19:32.234248 ip-10-0-143-24 kubenswrapper[4488]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:19:32.234248 ip-10-0-143-24 kubenswrapper[4488]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:19:32.234248 ip-10-0-143-24 kubenswrapper[4488]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:19:32.236397 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236312 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.236397 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236397 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.236397 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236401 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236406 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236409 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236412 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236415 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236418 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236420 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236423 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236425 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236428 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236430 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236433 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236435 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236441 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236444 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236447 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236449 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236452 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236454 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236457 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.236508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236459 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236462 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236464 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236467 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236469 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236472 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236474 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236477 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236480 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236482 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236486 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236491 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236494 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236497 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236500 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236502 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236505 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236508 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236510 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.237010 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236513 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236515 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236518 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236520 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236523 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236525 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236529 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236531 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236534 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236536 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236539 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236541 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236544 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236546 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236549 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236552 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236554 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236557 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236560 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236562 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.237505 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236565 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236567 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236569 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236572 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236575 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236578 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236580 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236583 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236585 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236588 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236591 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236594 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236596 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236599 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236602 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236604 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236607 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236610 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236613 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236615 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.237989 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236618 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236620 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236623 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236625 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236631 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236635 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236637 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236643 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236645 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236648 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236651 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236653 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236656 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236659 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236661 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236664 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236666 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236669 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236671 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.238469 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236674 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236678 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236682 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236684 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236687 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.236689 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237509 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237517 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237521 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237532 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237535 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237539 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237543 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237547 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237549 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237552 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237555 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237561 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.239048 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237564 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237566 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237569 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237572 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237574 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237577 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237579 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237581 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237584 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237586 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237589 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237591 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237593 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237596 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237598 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237600 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237603 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237605 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237607 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237610 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.239508 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237612 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237615 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237617 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237619 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237622 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237624 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237627 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237630 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237633 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237636 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237638 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237641 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237643 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237645 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237648 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237650 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237653 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237655 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237658 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.240074 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237660 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237662 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237665 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237667 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237670 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237672 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237675 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237677 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237679 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237682 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237684 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237686 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237689 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237691 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237693 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237696 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237698 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237701 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237703 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237706 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.240565 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237709 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237711 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237714 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237716 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237718 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237721 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237723 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237726 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237728 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237731 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237735 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237738 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237742 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237745 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237748 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237751 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237754 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237756 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237758 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237761 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.241078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237763 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237765 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237768 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237770 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237772 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237775 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237777 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237779 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237781 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237784 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237789 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237792 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237795 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237798 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.237800 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238307 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238313 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238316 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238318 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238321 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.241604 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238323 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238326 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238328 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238331 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238333 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238335 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238338 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238343 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238346 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238349 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238352 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238355 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238357 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238360 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238362 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238365 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238367 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238370 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238372 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.242078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238375 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238380 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238382 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238386 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238389 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238392 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238394 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238397 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238399 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238401 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238404 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238406 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238409 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238411 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238413 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238416 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238418 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238420 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238423 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.242580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238425 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238429 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238433 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238436 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238439 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238441 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238443 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238446 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238448 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238451 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238453 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238455 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238458 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238460 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238464 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238466 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238470 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238473 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238476 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.243025 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238478 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238481 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238484 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238486 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238489 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238491 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238494 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238496 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238498 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238501 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238503 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238506 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238508 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238511 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238513 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238515 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238518 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238520 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238523 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238525 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.243801 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238527 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238530 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238534 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238536 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238538 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238541 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238543 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238547 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238549 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238553 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238555 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238558 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238561 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238563 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238565 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238568 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238570 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238572 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238575 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238577 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238579 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.244579 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238582 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238584 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.238587 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238692 4488 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238712 4488 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238720 4488 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238726 4488 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238732 4488 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238735 4488 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238740 4488 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238744 4488 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238747 4488 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238750 4488 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238754 4488 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238757 4488 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238760 4488 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238763 4488 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238766 4488 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238771 4488 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238774 4488 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238779 4488 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238783 4488 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238786 4488 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:19:32.245344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238789 4488 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238793 4488 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238797 4488 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238801 4488 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238803 4488 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238806 4488 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238810 4488 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238813 4488 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238816 4488 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238819 4488 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238822 4488 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238825 4488 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238829 4488 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238832 4488 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238835 4488 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238838 4488 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238841 4488 flags.go:64] FLAG: --enable-server="true" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238844 4488 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238847 4488 flags.go:64] FLAG: --event-burst="100" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238851 4488 flags.go:64] FLAG: --event-qps="50" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238854 4488 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238856 4488 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238859 4488 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238864 4488 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:19:32.245876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238866 4488 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238869 4488 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238873 4488 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238877 4488 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238880 4488 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238885 4488 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238888 4488 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238890 4488 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238893 4488 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238896 4488 flags.go:64] FLAG: --feature-gates="" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238901 4488 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238904 4488 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238907 4488 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238910 4488 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238913 4488 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238916 4488 flags.go:64] FLAG: --help="false" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238919 4488 flags.go:64] FLAG: --hostname-override="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238923 4488 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238925 4488 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238928 4488 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238931 4488 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238935 4488 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238938 4488 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:19:32.246469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238941 4488 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238944 4488 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238947 4488 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238950 4488 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238953 4488 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238956 4488 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238958 4488 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238961 4488 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238964 4488 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238967 4488 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238970 4488 flags.go:64] FLAG: --lock-file="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238972 4488 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238977 4488 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238980 4488 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238987 4488 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238989 4488 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238992 4488 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238996 4488 flags.go:64] FLAG: --logging-format="text" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.238998 4488 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239002 4488 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239005 4488 flags.go:64] FLAG: --manifest-url="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239008 4488 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239012 4488 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239015 4488 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239019 4488 flags.go:64] FLAG: --max-pods="110" Oct 31 03:19:32.247023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239022 4488 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239025 4488 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239028 4488 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239031 4488 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239034 4488 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239037 4488 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239040 4488 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239047 4488 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239050 4488 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239053 4488 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239056 4488 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239059 4488 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239062 4488 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239066 4488 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239068 4488 flags.go:64] FLAG: --port="10250" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239071 4488 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239074 4488 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0eb8281b2cccee03e" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239078 4488 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239080 4488 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239100 4488 flags.go:64] FLAG: --register-node="true" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239103 4488 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239108 4488 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239111 4488 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239114 4488 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239117 4488 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239121 4488 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:19:32.247645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239124 4488 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239127 4488 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239130 4488 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239133 4488 flags.go:64] FLAG: --runonce="false" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239136 4488 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239139 4488 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239142 4488 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239145 4488 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239148 4488 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239151 4488 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239154 4488 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239157 4488 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239160 4488 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239163 4488 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239165 4488 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239168 4488 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239171 4488 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239173 4488 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239176 4488 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239182 4488 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239185 4488 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239187 4488 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239192 4488 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239195 4488 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239197 4488 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:19:32.248318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239201 4488 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239204 4488 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239207 4488 flags.go:64] FLAG: --v="2" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239211 4488 flags.go:64] FLAG: --version="false" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239215 4488 flags.go:64] FLAG: --vmodule="" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239219 4488 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239222 4488 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239365 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239369 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239372 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239377 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239380 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239382 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239385 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239387 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239390 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239392 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239395 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239397 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239400 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239402 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.248973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239404 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239408 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239412 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239415 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239418 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239421 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239424 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239426 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239429 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239432 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239434 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239438 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239440 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239442 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239445 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239447 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239451 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239453 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239456 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.249580 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239458 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239460 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239463 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239466 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239468 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239470 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239473 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239475 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239478 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239480 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239483 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239485 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239487 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239490 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239492 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239495 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239497 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239499 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239502 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239504 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.250265 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239507 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239509 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239512 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239514 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239518 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239521 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239523 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239526 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239528 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239530 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239533 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239535 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239538 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239540 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239542 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239545 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239548 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239553 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239556 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239560 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.251003 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239562 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239564 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239567 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239569 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239572 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239574 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239576 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239579 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239581 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239584 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239586 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239588 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239591 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239594 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239597 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239600 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239604 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239606 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239609 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.251617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239611 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239614 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239616 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239618 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239621 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239624 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239626 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239628 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239631 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239633 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239639 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239642 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239646 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239648 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.239674 4488 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239734 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.252203 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239738 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239742 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239744 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239747 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239750 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239753 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239755 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239759 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239761 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239764 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239766 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239769 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239771 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239774 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239777 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239780 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239783 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239785 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239788 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239790 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.252610 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239792 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239795 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239797 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239800 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239802 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239805 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239807 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239810 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239814 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239816 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239820 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239823 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239826 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239829 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239831 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239833 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239836 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239838 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239840 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.253280 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239843 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239845 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239848 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239850 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239853 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239855 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239867 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239872 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239874 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239877 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239879 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239882 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239884 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239886 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239889 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239891 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239913 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239915 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239917 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239920 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.253834 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239922 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239930 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239933 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239936 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239938 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239941 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239945 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239948 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239951 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239954 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239957 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239959 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239962 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239964 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239967 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239969 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239972 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239974 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239977 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.254391 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239983 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239985 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239988 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239990 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239993 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239995 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.239998 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240000 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240003 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240005 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240008 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240011 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240013 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240016 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240019 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240022 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240025 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240027 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240030 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240033 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.254846 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240036 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240038 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240041 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240043 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240045 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240048 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.240050 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.245584 4488 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.245599 4488 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.245618 4488 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.245623 4488 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245684 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245690 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245693 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245696 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245699 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245701 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245704 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245706 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.255332 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245709 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245711 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245713 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245716 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245718 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245721 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245723 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245726 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245729 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245732 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245735 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245737 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245739 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245742 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245744 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245747 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245749 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245767 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245771 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245774 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.255779 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245778 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245780 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245783 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245785 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245788 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245790 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245793 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245795 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245798 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245800 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245803 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245805 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245808 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245810 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245813 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245815 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245818 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245820 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245823 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245826 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245828 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.256302 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245831 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245833 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245836 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245838 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245841 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245845 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245849 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245852 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245855 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245858 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245860 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245863 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245865 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245867 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245869 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245872 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245874 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245876 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245879 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.256827 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245882 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245885 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245887 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245890 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245892 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245895 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245897 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245900 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245902 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245904 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245907 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245910 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245913 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245915 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245918 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245921 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245923 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245926 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245928 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.257407 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245930 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245933 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245935 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245937 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245941 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245945 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245948 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245950 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245953 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245956 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245958 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245961 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245963 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245966 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245968 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245970 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245973 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245975 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.257884 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.245977 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.246004 4488 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246167 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246174 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246178 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246181 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246184 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246188 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246191 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246193 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246196 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246198 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246201 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246204 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246206 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246209 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.258325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246211 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246214 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246216 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246219 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246223 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246225 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246228 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246230 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246232 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246235 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246237 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246240 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246242 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246244 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246247 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246249 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246251 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246254 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246256 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.258708 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246258 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246265 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246267 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246270 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246272 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246275 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246278 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246280 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246283 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246285 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246288 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246290 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246293 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246295 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246298 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246300 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246302 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246305 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246307 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.259197 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246310 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246312 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246314 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246317 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246319 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246323 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246326 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246329 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246332 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246335 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246338 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246340 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246343 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246345 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246349 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246352 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246354 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246356 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246361 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.259638 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246364 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246366 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246368 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246371 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246373 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246376 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246378 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246382 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246385 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246388 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246391 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246394 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246396 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246399 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246401 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246403 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246406 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246408 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246411 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246413 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.260308 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246415 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246418 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246420 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246422 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246425 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246427 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246430 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246434 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246436 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246439 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246441 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246445 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246448 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246451 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246453 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.246476 4488 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:19:32.261077 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246510 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246515 4488 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246517 4488 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246520 4488 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246523 4488 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246525 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246527 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246530 4488 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246532 4488 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246536 4488 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246540 4488 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246543 4488 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246546 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246549 4488 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246552 4488 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246554 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246558 4488 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246561 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246564 4488 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:19:32.261514 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246566 4488 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246569 4488 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246571 4488 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246574 4488 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246577 4488 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246579 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246582 4488 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246584 4488 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246587 4488 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246592 4488 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246594 4488 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246598 4488 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246602 4488 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246605 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246607 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246610 4488 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246612 4488 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246615 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246617 4488 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:19:32.261965 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246620 4488 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246622 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246624 4488 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246627 4488 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246629 4488 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246632 4488 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246634 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246637 4488 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246639 4488 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246642 4488 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246644 4488 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246646 4488 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246649 4488 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246651 4488 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246654 4488 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246656 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246658 4488 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246661 4488 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246663 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246666 4488 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246668 4488 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:19:32.262442 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246670 4488 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246674 4488 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246677 4488 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246679 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246682 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246685 4488 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246687 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246690 4488 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246692 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246695 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246697 4488 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246700 4488 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246702 4488 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246704 4488 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246707 4488 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246709 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246712 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246714 4488 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246717 4488 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:19:32.262939 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246719 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246722 4488 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246724 4488 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246726 4488 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246729 4488 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246731 4488 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246733 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246736 4488 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246738 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246740 4488 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246743 4488 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246745 4488 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246747 4488 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246750 4488 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246754 4488 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246756 4488 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246759 4488 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246762 4488 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246765 4488 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246767 4488 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:19:32.263400 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246770 4488 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246772 4488 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246774 4488 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246777 4488 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246779 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246782 4488 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246784 4488 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:32.246787 4488 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.246837 4488 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.247786 4488 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.247838 4488 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.248047 4488 server.go:1014] "Starting client certificate rotation" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.248205 4488 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.248257 4488 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:13:52 +0000 UTC" deadline="2028-05-02 17:51:18.704173035 +0000 UTC" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.248285 4488 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13190h31m46.455889447s" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.248516 4488 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:19:32.263874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.249622 4488 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:19:32.264370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.251421 4488 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:19:32.264370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.253127 4488 log.go:25] "Validated CRI v1 image API" Oct 31 03:19:32.264370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.253913 4488 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:19:32.264370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.255557 4488 fs.go:135] Filesystem UUIDs: map[56076a89-e5fc-4be6-b5b3-b849a6c05132:/dev/nvme0n1p4 629f963b-e4a2-47be-be6f-f79868eed778:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 03:19:32.264536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.255576 4488 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c864de68ffb1d7aa1ecc8fef481a71dcef884c627cf8f7e302a12c0261541d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c864de68ffb1d7aa1ecc8fef481a71dcef884c627cf8f7e302a12c0261541d3/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d2279f348c7640fe269f6f8d00eb915d8a22baa73518e0b28eb24cfdf32835b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d2279f348c7640fe269f6f8d00eb915d8a22baa73518e0b28eb24cfdf32835b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volumes/kubernetes.io~projected/kube-api-access-snvhk:{mountpoint:/var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volumes/kubernetes.io~projected/kube-api-access-snvhk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20d028be-4c41-438b-86ac-2e2eadce25df/volumes/kubernetes.io~projected/kube-api-access-p4l97:{mountpoint:/var/lib/kubelet/pods/20d028be-4c41-438b-86ac-2e2eadce25df/volumes/kubernetes.io~projected/kube-api-access-p4l97 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/449c2d15-e376-4ba5-a668-9c37ea713e3f/volumes/kubernetes.io~projected/kube-api-access-gfg98:{mountpoint:/var/lib/kubelet/pods/449c2d15-e376-4ba5-a668-9c37ea713e3f/volumes/kubernetes.io~projected/kube-api-access-gfg98 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56e24d88-5120-43d1-9649-9245d1967788/volumes/kubernetes.io~projected/kube-api-access-f8th4:{mountpoint:/var/lib/kubelet/pods/56e24d88-5120-43d1-9649-9245d1967788/volumes/kubernetes.io~projected/kube-api-access-f8th4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bf3c83e-4db7-4844-8aca-c66569729d3b/volumes/kubernetes.io~projected/kube-api-access-5shfs:{mountpoint:/var/lib/kubelet/pods/8bf3c83e-4db7-4844-8aca-c66569729d3b/volumes/kubernetes.io~projected/kube-api-access-5shfs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bf3c83e-4db7-4844-8aca-c66569729d3b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8bf3c83e-4db7-4844-8aca-c66569729d3b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98836313-6e91-46d0-84b9-f4306b05a80d/volumes/kubernetes.io~projected/kube-api-access-dtxxj:{mountpoint:/var/lib/kubelet/pods/98836313-6e91-46d0-84b9-f4306b05a80d/volumes/kubernetes.io~projected/kube-api-access-dtxxj major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9890fd99-71b5-45ab-a4f0-38293eff7802/volumes/kubernetes.io~projected/kube-api-access-rg9l5:{mountpoint:/var/lib/kubelet/pods/9890fd99-71b5-45ab-a4f0-38293eff7802/volumes/kubernetes.io~projected/kube-api-access-rg9l5 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9de23b18-25f8-43c3-9463-152099741b6b/volumes/kubernetes.io~projected/kube-api-access-zd8hv:{mountpoint:/var/lib/kubelet/pods/9de23b18-25f8-43c3-9463-152099741b6b/volumes/kubernetes.io~projected/kube-api-access-zd8hv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae4c6a55-1343-49ab-b514-78564e0f1d75/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ae4c6a55-1343-49ab-b514-78564e0f1d75/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~projected/kube-api-access-f7zjq:{mountpoint:/var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~projected/kube-api-access-f7zjq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c436b9f3-e0e4-4662-98bf-a140ef5c010c/volumes/kubernetes.io~projected/kube-api-access-5rj4z:{mountpoint:/var/lib/kubelet/pods/c436b9f3-e0e4-4662-98bf-a140ef5c010c/volumes/kubernetes.io~projected/kube-api-access-5rj4z major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d12ed103-14a9-4bf7-a1f6-3ccdf802838a/volumes/kubernetes.io~projected/kube-api-access-c84lg:{mountpoint:/var/lib/kubelet/pods/d12ed103-14a9-4bf7-a1f6-3ccdf802838a/volumes/kubernetes.io~projected/kube-api-access-c84lg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d66b1354-90f5-43ec-b395-cabd3dc171ac/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d66b1354-90f5-43ec-b395-cabd3dc171ac/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcebf912-082c-462b-80d4-aba939d35610/volumes/kubernetes.io~projected/kube-api-access-t4nfr:{mountpoint:/var/lib/kubelet/pods/dcebf912-082c-462b-80d4-aba939d35610/volumes/kubernetes.io~projected/kube-api-access-t4nfr major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de3f9807-0638-4b57-b00a-bb9bc112505b/volumes/kubernetes.io~projected/kube-api-access-t7ttk:{mountpoint:/var/lib/kubelet/pods/de3f9807-0638-4b57-b00a-bb9bc112505b/volumes/kubernetes.io~projected/kube-api-access-t7ttk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1/volumes/kubernetes.io~projected/kube-api-access-xzrk8:{mountpoint:/var/lib/kubelet/pods/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1/volumes/kubernetes.io~projected/kube-api-access-xzrk8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62/volumes/kubernetes.io~projected/kube-api-access-dqgrw:{mountpoint:/var/lib/kubelet/pods/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62/volumes/kubernetes.io~projected/kube-api-access-dqgrw major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a79f18ae986ccbd3dc89d3959505cd0b66e91b3a84262e1ef0f4e552ea92c573/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6fcf62f6f5860a94ec39b040bc3410b4b7211047b0b61f3036973080e04d1522/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/12ac06526879a17785a1b8c489a548ee6f6afd9c6c538b709cb4342f49c4df58/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a2bfb8b59aa5b431617661c1a2113967f5909ca5cab6e5250a9f733b35f40403/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f3f84afc5b6a718f9166c5c64a540eb280d094ddb4524ea56660e2f5776281fb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2d96641d25aa8b1e776f4e31428e69a68ad520c08678d99a027cbf32210de64e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/85fb4e8d7f7cddb5eff432c6484962a6fa4088f5d4312814d0936b94b323d429/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8c5cea0fe69acad6eb9d5573dd5804b1bdbfb48d976d72c1ead27a073e782986/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ed8cc6d8266d520e1bddc03e60dc91b155033907b9be23b83e2db13aa25a3570/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/da1c2d4be728fc19419ebfde896ad6454b0d93bf74777c5e72069d5aae163a1a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/f4ac2b792214da552abd63d4c6e690b7f65e0e6532b992dc0f934a4d3a865929/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/587a8990e982dbd05c895b4ad2a06305d45c7ef1d05496cc5013afd74121c01b/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e436059de0113231f72c0e8b987efe70fc93a2442b09d19f20c6dc21b52b2308/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e4fcf869d8a16ac1d74239c81227db885935d62529e3a7dffd913e3a6cdd5b4f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d40fe0c3da0080e607618c81cbb80c50325c0e3d45efe12eaf39dd5fcc6c2cf2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/94e1c84249ea6fb6130809e9cc3d208b1550dc8766411b50dba1b051c03fe74d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6ee0b1807a667b7fe47faab6b8bff614c5fa3b9f97c0a478321a582ae3a16a54/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/843e7be4e8aaddb022a5c07e4852fb3dc3d300af7fd31d4c46ad9f367c25a502/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/03733f315956d682de15a0d49c20638be742ecf13f9defd1e0025e03cdf2ec72/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7a395ce97039b300e3b49de50d6326bf31e68ac8c91fecc36ae3e1bbc7a86e32/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/89bae9a83c03b3deb1ef84f06a061a3faecbab7ff95749c8fcdf813d00ee40cc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/4f2ba929b56b49bdb92419224b50e49e0629075c2207f85dcfd69aef3e624238/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0091c6afe18d0471692529060d5cae26a035a61cad625e7c33164ed8bd5bb70a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f691f9c3e06d848962fc4e7c7bb9dd66cedd81e210f89aa1131ea4d202f36ad3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9c72ddb5c2bd2fb6d54feba24779e574ebcde7caa4109e194b87c010cee9e831/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/19425ce1c16826e8d4ed17aa57966263977686f3d9e74b9bb99305da8ec8c8b8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/0ab2cf169b1bc8633d0667a859f9253e614de49da218892ecf236bf12abd2095/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/eaa728c3f92382685f197285c32241604f903c76ca6428a04daa6a2d43e7edcc/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/686e057b9ee2afc2f404d6864d38a8cd35cc8f703855b98aba54c11f00d9bd67/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/87d67c8a03ec8b47627c68bc5952035d05e3b06e1b64ae58fba5aa8803801d11/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/495c246f908abeaf17cd9b572ca630d4972227d53f83bfb6779430590b5fc57e/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8644dc358acf1fb37f65c22890dc6e622a808c29709eee59b6dc3821ce275f55/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/70984937e2d479b30078d1a5169e25f7d9bb43e4fde0ffb6263b936b2b4e0881/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/cb678a97ce5433fe48dcaf780d584762c926aae38a745eaa100df6a5f310fd9e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/be8167b906c02fbd2d0323124ff6d292c46ea6f627ae06a7cada69d6b600995d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c423e55ea12b24a50035caed4e7c44de1df720fa7a928b64d797ceae569e7182/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/80cc6e8e4acefda65e58cfb62c4be48f3cdbe669796f6fa3c41bfa1bee6fb0ed/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/fe938efb916f842dc1b0347e1782c9321be6a80e26903de74cab1ff762a6fc4d/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/653108f6a1fcc90f38769d86f025be136e865a599e2cbfc671af7c7e9e6a0cdf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2a05cd98594e342714a91ce57a6571599016e7707d872a3a2e003b1153e2c77b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e9c47d65fda63524d3ec9c6c2b61954dbe284814d08a34bba1c1b90f591b20e5/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.262884 4488 manager.go:217] Machine: {Timestamp:2026-10-31 03:19:32.262431783 +0000 UTC m=+0.059150032 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200036 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282078128c76aaecef6ff181b05792 SystemUUID:ec282078-128c-76aa-ecef-6ff181b05792 BootID:88ca29bc-5031-4ac5-8ba7-7146d4790799 Filesystems:[{Device:/var/lib/kubelet/pods/9de23b18-25f8-43c3-9463-152099741b6b/volumes/kubernetes.io~projected/kube-api-access-zd8hv DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/449c2d15-e376-4ba5-a668-9c37ea713e3f/volumes/kubernetes.io~projected/kube-api-access-gfg98 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f5be1ecb1c73e5a0ce3dc2a937f94285671d2915a0f5da098cf2155db9793c9/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d2279f348c7640fe269f6f8d00eb915d8a22baa73518e0b28eb24cfdf32835b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c436b9f3-e0e4-4662-98bf-a140ef5c010c/volumes/kubernetes.io~projected/kube-api-access-5rj4z DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~projected/kube-api-access-f7zjq DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/614fa3b1ecca19efdb4ce77c1420c24800ae8574f8e89e49ac3c4a46dba78842/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bf3c83e-4db7-4844-8aca-c66569729d3b/volumes/kubernetes.io~projected/kube-api-access-5shfs DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98836313-6e91-46d0-84b9-f4306b05a80d/volumes/kubernetes.io~projected/kube-api-access-dtxxj DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7f4d07b02d33b43521353ef02c3df86e1fc69078b74c172d769a862617ac781/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56e24d88-5120-43d1-9649-9245d1967788/volumes/kubernetes.io~projected/kube-api-access-f8th4 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8efba16d12c56c7c4a08e9b77dbf56c7b8417388d652e52e5e50f30f8928ffe4/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d66b1354-90f5-43ec-b395-cabd3dc171ac/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dcebf912-082c-462b-80d4-aba939d35610/volumes/kubernetes.io~projected/kube-api-access-t4nfr DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae4c6a55-1343-49ab-b514-78564e0f1d75/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bf3c83e-4db7-4844-8aca-c66569729d3b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fe846307fb3435a8d840191a0b106761ce6982882fced0974d91245b6e102d0/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a60ededda86264670ec1521592676619f09a8652f80937c162d93db762181e05/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa26113ec591a3979f94cebad78faf79b5d7514ca17c976b1760595027d3a5de/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/de3f9807-0638-4b57-b00a-bb9bc112505b/volumes/kubernetes.io~projected/kube-api-access-t7ttk DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62/volumes/kubernetes.io~projected/kube-api-access-dqgrw DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1/volumes/kubernetes.io~projected/kube-api-access-xzrk8 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/20d028be-4c41-438b-86ac-2e2eadce25df/volumes/kubernetes.io~projected/kube-api-access-p4l97 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c864de68ffb1d7aa1ecc8fef481a71dcef884c627cf8f7e302a12c0261541d3/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d12ed103-14a9-4bf7-a1f6-3ccdf802838a/volumes/kubernetes.io~projected/kube-api-access-c84lg DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f08efdfc8951d851e6545c984665333abbd6e1420df986c80e40fae1a2e06dee/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a890b9d1b37517b623fd445d4cb19a41ea02f3ae43ab2708d6c5e6e3b9b71ca5/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1345bf92591a1239aed46204c72828035b72bd18810428c5b8e7c506c66bcca6/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0c92072e-6213-4831-b222-039305a4d122/volumes/kubernetes.io~projected/kube-api-access-snvhk DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9890fd99-71b5-45ab-a4f0-38293eff7802/volumes/kubernetes.io~projected/kube-api-access-rg9l5 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05be46eaa74155c MacAddress:0e:68:7c:3f:f1:25 Speed:10000 Mtu:8901} {Name:a60ededda862646 MacAddress:36:dd:ae:7b:93:56 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:69:01:55:e6:1d Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:a2:d3:c2:9a:82 Speed:0 Mtu:8901} {Name:e7f4d07b02d33b4 MacAddress:c2:25:d0:f1:63:56 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:69:01:55:e6:1d Speed:0 Mtu:9001} {Name:f08efdfc8951d85 MacAddress:ba:72:b9:dd:75:84 Speed:10000 Mtu:8901} {Name:fa26113ec591a39 MacAddress:a6:0a:1d:6d:57:db Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:cb:10:7a:ee:7e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0a:d4:8c:89:d3:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263144 4488 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263190 4488 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263469 4488 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263496 4488 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-24.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263621 4488 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263628 4488 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263639 4488 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263660 4488 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263754 4488 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.263947 4488 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264058 4488 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264109 4488 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264122 4488 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264133 4488 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264141 4488 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:19:32.264676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264150 4488 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:19:32.265485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264733 4488 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:19:32.265485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.264743 4488 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:19:32.265485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.265262 4488 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:19:32.267278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267265 4488 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:19:32.267278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267278 4488 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:19:32.267391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267382 4488 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 03:19:32.267581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267570 4488 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:19:32.267610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267599 4488 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:19:32.267713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267704 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:19:32.267744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267718 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:19:32.267744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267728 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:19:32.267744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267738 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267745 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267753 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267775 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267786 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267792 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267798 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:19:32.267814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267815 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:19:32.268021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267826 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:19:32.268021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267851 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:19:32.268021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.267865 4488 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:19:32.271210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.271199 4488 server.go:1263] "Started kubelet" Oct 31 03:19:32.271301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.271258 4488 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:19:32.271829 ip-10-0-143-24 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:19:32.272492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.271291 4488 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:19:32.272492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.271877 4488 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:19:32.272492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.272262 4488 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:19:32.272690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.272539 4488 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.275081 4488 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.275615 4488 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.275769 4488 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.275784 4488 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.275891 4488 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.276579 4488 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:54 +0000 UTC" deadline="2028-07-16 06:48:04.507330952 +0000 UTC" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.276644 4488 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14979h28m32.230691827s" Oct 31 03:19:32.280150 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.276919 4488 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:19:32.281843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281806 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c92072e-6213-4831-b222-039305a4d122" volumeName="kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-env-overrides" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281843 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c92072e-6213-4831-b222-039305a4d122" volumeName="kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-script-lib" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281859 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9890fd99-71b5-45ab-a4f0-38293eff7802" volumeName="kubernetes.io/empty-dir/9890fd99-71b5-45ab-a4f0-38293eff7802-data-volume" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281871 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12ed103-14a9-4bf7-a1f6-3ccdf802838a" volumeName="kubernetes.io/empty-dir/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-tmp-dir" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281885 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" volumeName="kubernetes.io/projected/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-kube-api-access-xzrk8" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281896 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9890fd99-71b5-45ab-a4f0-38293eff7802" volumeName="kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281906 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0186cf5-c4bb-461d-90f7-d7c3f9bb9754" volumeName="kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-tuned" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281917 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0186cf5-c4bb-461d-90f7-d7c3f9bb9754" volumeName="kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-tmp" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281931 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0186cf5-c4bb-461d-90f7-d7c3f9bb9754" volumeName="kubernetes.io/projected/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-kube-api-access-f7zjq" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281943 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12ed103-14a9-4bf7-a1f6-3ccdf802838a" volumeName="kubernetes.io/projected/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-kube-api-access-c84lg" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281955 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" volumeName="kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 03:19:32.281967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281965 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" volumeName="kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281976 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" volumeName="kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.281987 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcebf912-082c-462b-80d4-aba939d35610" volumeName="kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282004 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de3f9807-0638-4b57-b00a-bb9bc112505b" volumeName="kubernetes.io/projected/de3f9807-0638-4b57-b00a-bb9bc112505b-kube-api-access-t7ttk" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282015 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" volumeName="kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282027 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" volumeName="kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282038 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c92072e-6213-4831-b222-039305a4d122" volumeName="kubernetes.io/projected/0c92072e-6213-4831-b222-039305a4d122-kube-api-access-snvhk" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282049 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae4c6a55-1343-49ab-b514-78564e0f1d75" volumeName="kubernetes.io/configmap/ae4c6a55-1343-49ab-b514-78564e0f1d75-konnectivity-ca" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282060 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c436b9f3-e0e4-4662-98bf-a140ef5c010c" volumeName="kubernetes.io/projected/c436b9f3-e0e4-4662-98bf-a140ef5c010c-kube-api-access-5rj4z" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282073 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20d028be-4c41-438b-86ac-2e2eadce25df" volumeName="kubernetes.io/projected/20d028be-4c41-438b-86ac-2e2eadce25df-kube-api-access-p4l97" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282083 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e24d88-5120-43d1-9649-9245d1967788" volumeName="kubernetes.io/projected/56e24d88-5120-43d1-9649-9245d1967788-kube-api-access-f8th4" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282124 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12ed103-14a9-4bf7-a1f6-3ccdf802838a" volumeName="kubernetes.io/configmap/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-config-volume" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282135 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d66b1354-90f5-43ec-b395-cabd3dc171ac" volumeName="kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282146 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20d028be-4c41-438b-86ac-2e2eadce25df" volumeName="kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-cni-binary-copy" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282158 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449c2d15-e376-4ba5-a668-9c37ea713e3f" volumeName="kubernetes.io/projected/449c2d15-e376-4ba5-a668-9c37ea713e3f-kube-api-access-gfg98" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282170 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98836313-6e91-46d0-84b9-f4306b05a80d" volumeName="kubernetes.io/projected/98836313-6e91-46d0-84b9-f4306b05a80d-kube-api-access-dtxxj" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282181 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c92072e-6213-4831-b222-039305a4d122" volumeName="kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-config" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282203 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c92072e-6213-4831-b222-039305a4d122" volumeName="kubernetes.io/secret/0c92072e-6213-4831-b222-039305a4d122-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282212 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e24d88-5120-43d1-9649-9245d1967788" volumeName="kubernetes.io/configmap/56e24d88-5120-43d1-9649-9245d1967788-iptables-alerter-script" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282220 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf3c83e-4db7-4844-8aca-c66569729d3b" volumeName="kubernetes.io/projected/8bf3c83e-4db7-4844-8aca-c66569729d3b-kube-api-access-5shfs" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282228 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf3c83e-4db7-4844-8aca-c66569729d3b" volumeName="kubernetes.io/secret/8bf3c83e-4db7-4844-8aca-c66569729d3b-cert" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282237 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9de23b18-25f8-43c3-9463-152099741b6b" volumeName="kubernetes.io/configmap/9de23b18-25f8-43c3-9463-152099741b6b-serviceca" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282244 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" volumeName="kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-binary-copy" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282252 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" volumeName="kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282259 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20d028be-4c41-438b-86ac-2e2eadce25df" volumeName="kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-multus-daemon-config" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282267 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9890fd99-71b5-45ab-a4f0-38293eff7802" volumeName="kubernetes.io/projected/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-api-access-rg9l5" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282273 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9de23b18-25f8-43c3-9463-152099741b6b" volumeName="kubernetes.io/projected/9de23b18-25f8-43c3-9463-152099741b6b-kube-api-access-zd8hv" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282280 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae4c6a55-1343-49ab-b514-78564e0f1d75" volumeName="kubernetes.io/secret/ae4c6a55-1343-49ab-b514-78564e0f1d75-agent-certs" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282292 4488 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c436b9f3-e0e4-4662-98bf-a140ef5c010c" volumeName="kubernetes.io/empty-dir/c436b9f3-e0e4-4662-98bf-a140ef5c010c-tmp-dir" seLinuxMountContext="" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282300 4488 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:19:32.282498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.282305 4488 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283448 4488 factory.go:153] Registering CRI-O factory Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283465 4488 factory.go:223] Registration of the crio container factory successfully Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283522 4488 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283533 4488 factory.go:55] Registering systemd factory Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283540 4488 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283560 4488 factory.go:103] Registering Raw factory Oct 31 03:19:32.283745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283573 4488 manager.go:1199] Started watching for new ooms in manager Oct 31 03:19:32.283985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.283948 4488 manager.go:319] Starting recovery of all containers Oct 31 03:19:32.306993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.306956 4488 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:19:32.308675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.308655 4488 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:19:32.308675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.308673 4488 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:19:32.308863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.308695 4488 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:19:32.308863 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.308733 4488 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:19:32.333928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.333891 4488 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="7a371b0449780c7babd4385d7253069c42b66e5244603372968745ae463a0c0d" exitCode=0 Oct 31 03:19:32.333928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.333924 4488 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="31ac82a13e8d749fc64531c690e7057bf617d1db9f08c88f471457b945001bce" exitCode=0 Oct 31 03:19:32.333928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.333932 4488 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="65bcbbe57d26ae259a709beca920a23a905d58beb3a0bbf9bf806cd5d3299902" exitCode=0 Oct 31 03:19:32.333928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.333938 4488 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="59fd3eb0a1423e073b839c8039199ca8574a405dc0a0eea5483ae64f380ff9e1" exitCode=0 Oct 31 03:19:32.334264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.333947 4488 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="5f05aa69816841d897034c026218a249cf62c8f09e755817292a1f2de37f3e4a" exitCode=0 Oct 31 03:19:32.334264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.333956 4488 generic.go:356] "Generic (PLEG): container finished" podID="e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1" containerID="21cd0ce10c70a6eeb8662a6ba4b738fd83aa8b0de56968d3eb76b7665131a3cf" exitCode=0 Oct 31 03:19:32.339624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.339259 4488 generic.go:356] "Generic (PLEG): container finished" podID="0cd30d86732a0a5d7b8493f1d7958e8b" containerID="bc96389109a4d5899d3029fd619e07c1037b66a695467c1daefb29c5b3ac9c2a" exitCode=0 Oct 31 03:19:32.350674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.350653 4488 manager.go:324] Recovery completed Oct 31 03:19:32.363275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.363254 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.364573 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.364547 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.364655 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.364583 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.364655 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.364596 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.366017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366003 4488 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:19:32.366017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366015 4488 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:19:32.366121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366019 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.366121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366031 4488 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366168 4488 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366179 4488 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366195 4488 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366200 4488 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366206 4488 policy_none.go:50] "Start" Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366215 4488 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:19:32.366267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366224 4488 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:19:32.366451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366397 4488 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 03:19:32.366451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366405 4488 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 03:19:32.366451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366411 4488 policy_none.go:44] "Start" Oct 31 03:19:32.366863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366849 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.366900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366877 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.366900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.366888 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.371914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.371884 4488 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:19:32.376111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376075 4488 manager.go:341] "Starting Device Plugin manager" Oct 31 03:19:32.376214 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.376119 4488 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:19:32.376214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376130 4488 server.go:87] "Starting device plugin registration server" Oct 31 03:19:32.376421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376405 4488 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:19:32.376489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376421 4488 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:19:32.376536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376517 4488 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:19:32.376672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376625 4488 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:19:32.376728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376684 4488 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:19:32.376728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376689 4488 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:19:32.376821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.376746 4488 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:19:32.376670972Z","UUID":"24a82a5c-e37d-44ab-94a7-bc1bd027085c","Handler":null,"Name":"","Endpoint":""} Oct 31 03:19:32.378031 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.378007 4488 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:19:32.378179 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.378062 4488 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" Oct 31 03:19:32.378275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.378261 4488 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:19:32.378329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.378282 4488 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:19:32.409298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.409245 4488 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal"] Oct 31 03:19:32.409439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.409386 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.410414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.410398 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.410490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.410431 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.410490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.410442 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.410684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.410675 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.410841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.410827 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.411443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411430 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.411482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411450 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.411482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411458 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.411482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411471 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.411482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411473 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.411482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411482 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.411645 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.411633 4488 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.411775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411742 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"0cd30d86732a0a5d7b8493f1d7958e8b","Type":"ContainerStarted","Data":"9a08c1f524325de640dddc097374ec5120776bf31b2d543c2134d28a9e60a1f9"} Oct 31 03:19:32.411807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411783 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"0cd30d86732a0a5d7b8493f1d7958e8b","Type":"ContainerDied","Data":"bc96389109a4d5899d3029fd619e07c1037b66a695467c1daefb29c5b3ac9c2a"} Oct 31 03:19:32.411807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411795 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.411874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411797 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"0cd30d86732a0a5d7b8493f1d7958e8b","Type":"ContainerStarted","Data":"0c4c6d8cc312629c70fdc56eea1bbd8c3464a318d15e644039b317d7c71960ee"} Oct 31 03:19:32.411922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411890 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"abb3780775d238ba0d694f1e9cc7fb2a","Type":"ContainerStarted","Data":"04bebd4b09a044bf043a87bdabc043ecd869c15fa5cb6249ea964ca099b01eb6"} Oct 31 03:19:32.411957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.411924 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" event={"ID":"abb3780775d238ba0d694f1e9cc7fb2a","Type":"ContainerStarted","Data":"ef7d068606fdf566dafe7edd9a7790fd4ab809b96e96b61006439c303662c9c0"} Oct 31 03:19:32.412537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.412522 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.412638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.412549 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.412638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.412562 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.412703 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.412694 4488 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-24.us-west-1.compute.internal\" not found" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.477203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.477166 4488 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:19:32.477486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.477469 4488 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 03:19:32.478021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.478004 4488 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:32.478114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.478023 4488 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:32.478321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.478306 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:19:32.478375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.478338 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:19:32.478375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.478352 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 03:19:32.478443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.478385 4488 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.528416 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.528323 4488 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:32.528618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.528601 4488 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:19:32.577278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.577242 4488 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.578373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.578356 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.578414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.578390 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.578454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.578414 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb3780775d238ba0d694f1e9cc7fb2a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"abb3780775d238ba0d694f1e9cc7fb2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.628715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.628684 4488 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.628886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.628757 4488 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.629391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.629372 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeNotReady" Oct 31 03:19:32.629475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.629407 4488 setters.go:546] "Node became not ready" node="ip-10-0-143-24.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T03:19:32Z","lastTransitionTime":"2026-10-31T03:19:32Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 03:19:32.637777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.637759 4488 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:32.637828 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.637803 4488 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.637828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.637822 4488 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.642267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.642252 4488 nodeinfomanager.go:471] existing CSINode "ip-10-0-143-24.us-west-1.compute.internal" is owned by different node (oldNodeID="f32d701f-6b29-47fe-bbc9-ebbaf175eb8e", newNodeID=""), cleaning up... Oct 31 03:19:32.678579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.678546 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.678721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.678583 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.678721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.678620 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb3780775d238ba0d694f1e9cc7fb2a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"abb3780775d238ba0d694f1e9cc7fb2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.678721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.678659 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.678721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.678685 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd30d86732a0a5d7b8493f1d7958e8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"0cd30d86732a0a5d7b8493f1d7958e8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.678721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.678661 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb3780775d238ba0d694f1e9cc7fb2a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" (UID: \"abb3780775d238ba0d694f1e9cc7fb2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.700552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.700527 4488 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:32.700674 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:32.700575 4488 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:32.900810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.900783 4488 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-24.us-west-1.compute.internal" event="NodeReady" Oct 31 03:19:32.900955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:32.900860 4488 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:19:33.264719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.264640 4488 apiserver.go:52] "Watching apiserver" Oct 31 03:19:33.350339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.350310 4488 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:33.350507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.350443 4488 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:33.368925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.368894 4488 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:19:33.369379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.369360 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-r7w79","openshift-network-operator/iptables-alerter-cnthq","openshift-image-registry/node-ca-ftcg9","openshift-cluster-node-tuning-operator/tuned-bljwr","openshift-dns/dns-default-6bbvw","openshift-ingress-canary/ingress-canary-x5fwg","openshift-insights/periodic-gathering-9vj2z-6h7j2","openshift-ovn-kubernetes/ovnkube-node-ns5dr","kube-system/global-pull-secret-syncer-jvb8m","kube-system/konnectivity-agent-kms5j","openshift-multus/network-metrics-daemon-x75qg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79","kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-h4tqk","openshift-dns/node-resolver-vl82x","openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw","openshift-multus/multus-additional-cni-plugins-fpp2c","openshift-insights/insights-runtime-extractor-qs77w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal"] Oct 31 03:19:33.375504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.375469 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.376125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.376084 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.377349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.377324 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.378670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.378651 4488 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:19:33.381877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.381855 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.381984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.381893 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.381984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.381936 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bf3c83e-4db7-4844-8aca-c66569729d3b-cert\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:33.381984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.381959 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.382144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.381983 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c436b9f3-e0e4-4662-98bf-a140ef5c010c-hosts-file\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.382144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382007 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-ovn\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382031 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382081 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c92072e-6213-4831-b222-039305a4d122-ovn-node-metrics-cert\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382145 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvhk\" (UniqueName: \"kubernetes.io/projected/0c92072e-6213-4831-b222-039305a4d122-kube-api-access-snvhk\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382188 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rj4z\" (UniqueName: \"kubernetes.io/projected/c436b9f3-e0e4-4662-98bf-a140ef5c010c-kube-api-access-5rj4z\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382218 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-system-cni-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382239 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382268 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382293 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-systemd-units\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382317 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-node-log\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382334 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-cni-binary-copy\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.382521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382348 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84lg\" (UniqueName: \"kubernetes.io/projected/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-kube-api-access-c84lg\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.382521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382378 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shfs\" (UniqueName: \"kubernetes.io/projected/8bf3c83e-4db7-4844-8aca-c66569729d3b-kube-api-access-5shfs\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:33.382521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382439 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9890fd99-71b5-45ab-a4f0-38293eff7802-data-volume\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382536 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382556 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9890fd99-71b5-45ab-a4f0-38293eff7802-data-volume\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382565 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ttk\" (UniqueName: \"kubernetes.io/projected/de3f9807-0638-4b57-b00a-bb9bc112505b-kube-api-access-t7ttk\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382601 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56e24d88-5120-43d1-9649-9245d1967788-iptables-alerter-script\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.382734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382722 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.382764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382742 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-socket-dir-parent\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.382764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382758 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.382818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382772 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.382818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382790 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56e24d88-5120-43d1-9649-9245d1967788-host-slash\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.382869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382821 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-kubelet\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.382897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382878 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-k8s-cni-cncf-io\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.382897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382886 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.382960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382900 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-conf-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.382960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382931 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-etc-kubernetes\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.382960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382951 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9l5\" (UniqueName: \"kubernetes.io/projected/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-api-access-rg9l5\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.383042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.382976 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.383135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383029 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.383135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383064 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8hv\" (UniqueName: \"kubernetes.io/projected/9de23b18-25f8-43c3-9463-152099741b6b-kube-api-access-zd8hv\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.383135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383114 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-slash\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383132 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-systemd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383202 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-var-lib-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383257 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-socket-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.383301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383288 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxxj\" (UniqueName: \"kubernetes.io/projected/98836313-6e91-46d0-84b9-f4306b05a80d-kube-api-access-dtxxj\") pod \"migrator-66df5d496b-ffmpw\" (UID: \"98836313-6e91-46d0-84b9-f4306b05a80d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:33.383329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383311 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-bin\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383326 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-env-overrides\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383467 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383449 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-script-lib\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383481 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-registration-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.383510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383498 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-device-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.383590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383522 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-system-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.383590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383561 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-cnibin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.383590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383577 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9de23b18-25f8-43c3-9463-152099741b6b-serviceca\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.383678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383617 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.383678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383638 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-netns\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.383678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383652 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-var-lib-kubelet\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.383678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383671 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de23b18-25f8-43c3-9463-152099741b6b-host\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.383780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383700 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-netns\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383720 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-netd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.383780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383741 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysconfig\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.383780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383761 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-run\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.383911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383786 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-tuned\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.383911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383836 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-tuned\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.383911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383853 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-tmp\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.383911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383878 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zjq\" (UniqueName: \"kubernetes.io/projected/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-kube-api-access-f7zjq\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383911 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383926 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436b9f3-e0e4-4662-98bf-a140ef5c010c-tmp-dir\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383943 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-kubelet-config\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.383973 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-tmp\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384027 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436b9f3-e0e4-4662-98bf-a140ef5c010c-tmp-dir\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384035 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-config\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.384082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384082 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384131 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l97\" (UniqueName: \"kubernetes.io/projected/20d028be-4c41-438b-86ac-2e2eadce25df-kube-api-access-p4l97\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384195 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9890fd99-71b5-45ab-a4f0-38293eff7802-crio-socket\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384222 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrk8\" (UniqueName: \"kubernetes.io/projected/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-kube-api-access-xzrk8\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384273 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-etc-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384307 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384365 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-multus-daemon-config\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384398 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-multus-certs\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.384441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384428 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-config-volume\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384452 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384500 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-modprobe-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384526 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-conf\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384549 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-systemd\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384585 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-sys\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384632 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384661 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8th4\" (UniqueName: \"kubernetes.io/projected/56e24d88-5120-43d1-9649-9245d1967788-kube-api-access-f8th4\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384687 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfg98\" (UniqueName: \"kubernetes.io/projected/449c2d15-e376-4ba5-a668-9c37ea713e3f-kube-api-access-gfg98\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384713 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384738 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384763 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae4c6a55-1343-49ab-b514-78564e0f1d75-agent-certs\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384789 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-bin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384817 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-multus\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.384855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384844 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-kubelet\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-hostroot\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384901 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-tmp-dir\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384939 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-log-socket\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.384989 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-tmp-dir\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385005 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385047 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-sys-fs\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385081 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-kubernetes\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385135 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-lib-modules\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385168 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cnibin\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385190 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-dbus\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385217 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c6a55-1343-49ab-b514-78564e0f1d75-konnectivity-ca\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:33.385408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.385243 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-host\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.409680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.409652 4488 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:33.409791 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:33.409714 4488 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:33.410273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.410257 4488 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:19:33.410348 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:33.410302 4488 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-24.us-west-1.compute.internal" Oct 31 03:19:33.422542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.422511 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz7gs\"" Oct 31 03:19:33.423342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.423319 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4wd7m\"" Oct 31 03:19:33.423342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.423327 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltpjd\"" Oct 31 03:19:33.423598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.423381 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4ln2\"" Oct 31 03:19:33.423598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.423468 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vxflf\"" Oct 31 03:19:33.423598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.423505 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67ksm\"" Oct 31 03:19:33.423787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.423692 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.424055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424031 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:19:33.424184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424040 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.424184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424070 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:19:33.424184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424081 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.424184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424129 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:19:33.424184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424148 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:19:33.424184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424149 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424196 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424207 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424198 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424200 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424217 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424233 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424241 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.424406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.424396 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.433167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.433145 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56e24d88-5120-43d1-9649-9245d1967788-iptables-alerter-script\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.433167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.433149 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-cni-binary-copy\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.433289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.433155 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.433289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.433193 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.433289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.433149 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.434650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.434629 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9de23b18-25f8-43c3-9463-152099741b6b-serviceca\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.460262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.460231 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rj4z\" (UniqueName: \"kubernetes.io/projected/c436b9f3-e0e4-4662-98bf-a140ef5c010c-kube-api-access-5rj4z\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.462073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.462039 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8th4\" (UniqueName: \"kubernetes.io/projected/56e24d88-5120-43d1-9649-9245d1967788-kube-api-access-f8th4\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.462379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.462351 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8hv\" (UniqueName: \"kubernetes.io/projected/9de23b18-25f8-43c3-9463-152099741b6b-kube-api-access-zd8hv\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.462556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.462533 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84lg\" (UniqueName: \"kubernetes.io/projected/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-kube-api-access-c84lg\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.462790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.462777 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nfr\" (UniqueName: \"kubernetes.io/projected/dcebf912-082c-462b-80d4-aba939d35610-kube-api-access-t4nfr\") pod \"network-check-target-h4tqk\" (UID: \"dcebf912-082c-462b-80d4-aba939d35610\") " pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:33.467772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.467753 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zjq\" (UniqueName: \"kubernetes.io/projected/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-kube-api-access-f7zjq\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.485485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485457 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485488 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-socket-dir-parent\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485510 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.485592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485583 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56e24d88-5120-43d1-9649-9245d1967788-host-slash\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.485678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485600 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-socket-dir-parent\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485589 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485611 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-kubelet\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485632 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56e24d88-5120-43d1-9649-9245d1967788-host-slash\") pod \"iptables-alerter-cnthq\" (UID: \"56e24d88-5120-43d1-9649-9245d1967788\") " pod="openshift-network-operator/iptables-alerter-cnthq" Oct 31 03:19:33.485678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485644 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-kubelet\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485663 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-k8s-cni-cncf-io\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485683 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-conf-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485699 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-etc-kubernetes\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485722 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485739 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-multus-conf-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485746 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-slash\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485768 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-systemd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485778 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-k8s-cni-cncf-io\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485746 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-etc-kubernetes\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485787 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-var-lib-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485789 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485798 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-slash\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485812 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-socket-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485817 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-systemd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485830 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-var-lib-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.485840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485834 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-bin\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485864 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-registration-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485877 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-socket-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485871 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-bin\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485891 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-device-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485917 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-registration-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485924 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-system-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485949 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-cnibin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485972 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485929 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-device-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.485995 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-netns\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486003 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-system-cni-dir\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486040 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-netns\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486043 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-cnibin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486043 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486064 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-var-lib-kubelet\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486109 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de23b18-25f8-43c3-9463-152099741b6b-host\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486134 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-netns\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486136 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-var-lib-kubelet\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486158 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-netd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486183 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysconfig\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486183 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de23b18-25f8-43c3-9463-152099741b6b-host\") pod \"node-ca-ftcg9\" (UID: \"9de23b18-25f8-43c3-9463-152099741b6b\") " pod="openshift-image-registry/node-ca-ftcg9" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486198 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-netns\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486221 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-cni-netd\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486224 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysconfig\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486277 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-run\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486309 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-kubelet-config\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486328 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-run\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486336 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486379 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9890fd99-71b5-45ab-a4f0-38293eff7802-crio-socket\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486399 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-kubelet-config\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486401 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486408 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-etc-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486441 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9890fd99-71b5-45ab-a4f0-38293eff7802-crio-socket\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486445 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-etc-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486441 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.486437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486467 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-openvswitch\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486474 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-multus-certs\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486491 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486505 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-modprobe-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486520 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-conf\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486538 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-systemd\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486507 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-run-multus-certs\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486552 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-sys\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486593 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-bin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486605 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-systemd\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486621 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-multus\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486637 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-bin\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486642 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-modprobe-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486646 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-kubelet\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486661 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-cni-multus\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486660 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-sys\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486675 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-host-var-lib-kubelet\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486677 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-hostroot\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486693 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-conf\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486705 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20d028be-4c41-438b-86ac-2e2eadce25df-hostroot\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486702 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-log-socket\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486729 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486754 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486755 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-log-socket\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486763 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-sys-fs\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486782 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-kubernetes\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486796 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-lib-modules\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486812 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cnibin\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486835 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-dbus\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486856 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-kubernetes\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486859 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-host\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486869 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/449c2d15-e376-4ba5-a668-9c37ea713e3f-sys-fs\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486883 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-cnibin\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486903 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-host\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486913 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-lib-modules\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486919 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d66b1354-90f5-43ec-b395-cabd3dc171ac-dbus\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486890 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486969 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c436b9f3-e0e4-4662-98bf-a140ef5c010c-hosts-file\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.486994 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-ovn\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487016 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487046 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-system-cni-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487071 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487073 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-systemd-units\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487049 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c436b9f3-e0e4-4662-98bf-a140ef5c010c-hosts-file\") pod \"node-resolver-vl82x\" (UID: \"c436b9f3-e0e4-4662-98bf-a140ef5c010c\") " pod="openshift-dns/node-resolver-vl82x" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487080 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-run-ovn\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487105 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-system-cni-dir\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487125 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-node-log\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487138 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-systemd-units\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487184 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c92072e-6213-4831-b222-039305a4d122-node-log\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487203 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.487610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.487295 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0186cf5-c4bb-461d-90f7-d7c3f9bb9754-etc-sysctl-d\") pod \"tuned-bljwr\" (UID: \"c0186cf5-c4bb-461d-90f7-d7c3f9bb9754\") " pod="openshift-cluster-node-tuning-operator/tuned-bljwr" Oct 31 03:19:33.522134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.521946 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:19:33.522134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522013 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:19:33.522134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522054 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:19:33.522134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522128 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xmnbh\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522254 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bf3c83e-4db7-4844-8aca-c66569729d3b-cert\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522270 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jjf7q\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522415 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4kwqf\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522470 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9x6rv\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522500 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522504 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sv7k2\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522525 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-n6jvn\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522507 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:19:33.522536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522534 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522515 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b5nvx\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522552 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v5ddh\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522561 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522555 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dbgqm\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522578 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkxjs\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522560 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btzr7\"" Oct 31 03:19:33.522836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522562 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:19:33.523162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.522979 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:19:33.523162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523027 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.523162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523035 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.523162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523145 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.523368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523342 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:19:33.523445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523392 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d66b1354-90f5-43ec-b395-cabd3dc171ac-original-pull-secret\") pod \"global-pull-secret-syncer-jvb8m\" (UID: \"d66b1354-90f5-43ec-b395-cabd3dc171ac\") " pod="kube-system/global-pull-secret-syncer-jvb8m" Oct 31 03:19:33.523445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523411 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.523602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523501 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-env-overrides\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.523602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523510 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.523602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523551 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.523602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523574 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523623 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523675 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523686 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523703 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523708 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523721 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523724 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523757 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 03:19:33.524003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.523836 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:19:33.524375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.524066 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:19:33.524375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.524244 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-config\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.524580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.524566 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20d028be-4c41-438b-86ac-2e2eadce25df-multus-daemon-config\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.525331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.525319 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.525430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.525415 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-config-volume\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:33.525480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.525467 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c6a55-1343-49ab-b514-78564e0f1d75-konnectivity-ca\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:33.525529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.525515 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae4c6a55-1343-49ab-b514-78564e0f1d75-agent-certs\") pod \"konnectivity-agent-kms5j\" (UID: \"ae4c6a55-1343-49ab-b514-78564e0f1d75\") " pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:33.526127 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:33.526113 4488 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:33.526189 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:33.526181 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:34.026163385 +0000 UTC m=+1.822881621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:33.527227 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:33.527217 4488 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:33.527262 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:33.527253 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:34.027245428 +0000 UTC m=+1.823963665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:33.527422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.527407 4488 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:19:33.527702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.527686 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9l5\" (UniqueName: \"kubernetes.io/projected/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-api-access-rg9l5\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.527947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.527878 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfg98\" (UniqueName: \"kubernetes.io/projected/449c2d15-e376-4ba5-a668-9c37ea713e3f-kube-api-access-gfg98\") pod \"aws-ebs-csi-driver-node-tdg79\" (UID: \"449c2d15-e376-4ba5-a668-9c37ea713e3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tdg79" Oct 31 03:19:33.528230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.528212 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.529771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.529752 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3f9807-0638-4b57-b00a-bb9bc112505b-metrics-certs\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533252 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9890fd99-71b5-45ab-a4f0-38293eff7802-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533370 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c92072e-6213-4831-b222-039305a4d122-ovn-node-metrics-cert\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533481 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrk8\" (UniqueName: \"kubernetes.io/projected/e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1-kube-api-access-xzrk8\") pod \"multus-additional-cni-plugins-fpp2c\" (UID: \"e5e427ca-6b3a-4cf5-aff3-f0f7cdcf77b1\") " pod="openshift-multus/multus-additional-cni-plugins-fpp2c" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533478 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxxj\" (UniqueName: \"kubernetes.io/projected/98836313-6e91-46d0-84b9-f4306b05a80d-kube-api-access-dtxxj\") pod \"migrator-66df5d496b-ffmpw\" (UID: \"98836313-6e91-46d0-84b9-f4306b05a80d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ffmpw" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533528 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ttk\" (UniqueName: \"kubernetes.io/projected/de3f9807-0638-4b57-b00a-bb9bc112505b-kube-api-access-t7ttk\") pod \"network-metrics-daemon-x75qg\" (UID: \"de3f9807-0638-4b57-b00a-bb9bc112505b\") " pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533616 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l97\" (UniqueName: \"kubernetes.io/projected/20d028be-4c41-438b-86ac-2e2eadce25df-kube-api-access-p4l97\") pod \"multus-r7w79\" (UID: \"20d028be-4c41-438b-86ac-2e2eadce25df\") " pod="openshift-multus/multus-r7w79" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533743 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shfs\" (UniqueName: \"kubernetes.io/projected/8bf3c83e-4db7-4844-8aca-c66569729d3b-kube-api-access-5shfs\") pod \"ingress-canary-x5fwg\" (UID: \"8bf3c83e-4db7-4844-8aca-c66569729d3b\") " pod="openshift-ingress-canary/ingress-canary-x5fwg" Oct 31 03:19:33.533773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.533750 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c92072e-6213-4831-b222-039305a4d122-ovnkube-script-lib\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.534145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.534053 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvhk\" (UniqueName: \"kubernetes.io/projected/0c92072e-6213-4831-b222-039305a4d122-kube-api-access-snvhk\") pod \"ovnkube-node-ns5dr\" (UID: \"0c92072e-6213-4831-b222-039305a4d122\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:33.572032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.571999 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:19:33.575308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.575285 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") pod \"periodic-gathering-9vj2z-6h7j2\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:19:33.677786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.677753 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x75qg" Oct 31 03:19:33.787418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:33.787332 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:34.047446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.047213 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x75qg"] Oct 31 03:19:34.051507 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:34.051477 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3f9807_0638_4b57_b00a_bb9bc112505b.slice/crio-076b41ebec3ba53727ba50a91a900179e31f9061147b3d9ab2d12e0a0cacdd9a WatchSource:0}: Error finding container 076b41ebec3ba53727ba50a91a900179e31f9061147b3d9ab2d12e0a0cacdd9a: Status 404 returned error can't find the container with id 076b41ebec3ba53727ba50a91a900179e31f9061147b3d9ab2d12e0a0cacdd9a Oct 31 03:19:34.054149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.054132 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:19:34.091727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.091684 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:34.091960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.091749 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:34.091960 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:34.091837 4488 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:34.091960 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:34.091867 4488 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:34.091960 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:34.091930 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.091915092 +0000 UTC m=+2.888633342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:34.091960 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:34.091947 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:35.091940637 +0000 UTC m=+2.888658874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:34.354160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.354122 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x75qg" event={"ID":"de3f9807-0638-4b57-b00a-bb9bc112505b","Type":"ContainerStarted","Data":"076b41ebec3ba53727ba50a91a900179e31f9061147b3d9ab2d12e0a0cacdd9a"} Oct 31 03:19:34.391222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.391183 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:34.590608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.590574 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:34.993308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:34.993269 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:35.097707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:35.097674 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:35.097885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:35.097735 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:35.097885 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:35.097829 4488 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:35.097974 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:35.097890 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:37.097876216 +0000 UTC m=+4.894594464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:35.097974 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:35.097829 4488 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:35.097974 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:35.097949 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:37.097937683 +0000 UTC m=+4.894655922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:35.192651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:35.192616 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:35.793480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:35.793374 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:36.361081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:36.361041 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x75qg" event={"ID":"de3f9807-0638-4b57-b00a-bb9bc112505b","Type":"ContainerStarted","Data":"ad99d72032550842ef2f41db6943ab22e6ead24351fa84c269b7846a25d3d67a"} Oct 31 03:19:36.361255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:36.361123 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x75qg" event={"ID":"de3f9807-0638-4b57-b00a-bb9bc112505b","Type":"ContainerStarted","Data":"b1d9d703de72e38d9e8bb4ac6dc7ad4e3129a24fa782e36b545135a23c7236f8"} Oct 31 03:19:36.394367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:36.394332 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:36.596453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:36.596419 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:36.989173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:36.989141 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:37.109587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:37.109545 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:37.109756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:37.109604 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:37.109756 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:37.109703 4488 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 03:19:37.109756 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:37.109724 4488 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 03:19:37.109845 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:37.109780 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls podName:d12ed103-14a9-4bf7-a1f6-3ccdf802838a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:41.109765893 +0000 UTC m=+8.906484130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls") pod "dns-default-6bbvw" (UID: "d12ed103-14a9-4bf7-a1f6-3ccdf802838a") : secret "dns-default-metrics-tls" not found Oct 31 03:19:37.109845 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:19:37.109796 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls podName:9890fd99-71b5-45ab-a4f0-38293eff7802 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:41.109789282 +0000 UTC m=+8.906507518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qs77w" (UID: "9890fd99-71b5-45ab-a4f0-38293eff7802") : secret "insights-runtime-extractor-tls" not found Oct 31 03:19:37.139511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:37.139470 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:37.291382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:37.291305 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:37.793256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:37.793226 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:38.392463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:38.392430 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:38.589326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:38.589295 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:38.659172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:38.659076 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kms5j" Oct 31 03:19:38.744750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:38.744719 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:38.990219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:38.990130 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.190218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:39.190178 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.328401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:39.328375 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:39.347151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:39.347125 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.590605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:39.590527 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.766947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:39.766915 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.917243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:39.917159 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:40.072234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:40.072202 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:40.377965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:40.377938 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:40.378198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:40.378185 4488 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:19:40.821202 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:40.821123 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:40.999339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:40.999302 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:19:41.036991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.036957 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:41.135954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.135893 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:41.136137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.135975 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:41.138586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.138560 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d12ed103-14a9-4bf7-a1f6-3ccdf802838a-metrics-tls\") pod \"dns-default-6bbvw\" (UID: \"d12ed103-14a9-4bf7-a1f6-3ccdf802838a\") " pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:41.138704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.138560 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9890fd99-71b5-45ab-a4f0-38293eff7802-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qs77w\" (UID: \"9890fd99-71b5-45ab-a4f0-38293eff7802\") " pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:41.176558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.176531 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:41.176668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.176612 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qs77w" Oct 31 03:19:41.595483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.595452 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bbvw"] Oct 31 03:19:41.636628 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:41.636595 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12ed103_14a9_4bf7_a1f6_3ccdf802838a.slice/crio-43cd60941f6f00f10c0196d5990263e03bccf4c1794328719a54cd5a9baa1d53 WatchSource:0}: Error finding container 43cd60941f6f00f10c0196d5990263e03bccf4c1794328719a54cd5a9baa1d53: Status 404 returned error can't find the container with id 43cd60941f6f00f10c0196d5990263e03bccf4c1794328719a54cd5a9baa1d53 Oct 31 03:19:41.646448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:41.646422 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qs77w"] Oct 31 03:19:41.649641 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:41.649618 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9890fd99_71b5_45ab_a4f0_38293eff7802.slice/crio-21d8f89cc656983a1026469785c4d4fca819763bab8612a6e181465fb7ec85dd WatchSource:0}: Error finding container 21d8f89cc656983a1026469785c4d4fca819763bab8612a6e181465fb7ec85dd: Status 404 returned error can't find the container with id 21d8f89cc656983a1026469785c4d4fca819763bab8612a6e181465fb7ec85dd Oct 31 03:19:42.378396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:42.378352 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs77w" event={"ID":"9890fd99-71b5-45ab-a4f0-38293eff7802","Type":"ContainerStarted","Data":"47919c3764a71a2a9b4e731c255807f316b4e40fdd180ca12d4c2a910ca15295"} Oct 31 03:19:42.378903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:42.378409 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs77w" event={"ID":"9890fd99-71b5-45ab-a4f0-38293eff7802","Type":"ContainerStarted","Data":"21d8f89cc656983a1026469785c4d4fca819763bab8612a6e181465fb7ec85dd"} Oct 31 03:19:42.384065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:42.384038 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bbvw" event={"ID":"d12ed103-14a9-4bf7-a1f6-3ccdf802838a","Type":"ContainerStarted","Data":"43cd60941f6f00f10c0196d5990263e03bccf4c1794328719a54cd5a9baa1d53"} Oct 31 03:19:42.755788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:42.755719 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns5dr" Oct 31 03:19:43.390072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:43.390019 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs77w" event={"ID":"9890fd99-71b5-45ab-a4f0-38293eff7802","Type":"ContainerStarted","Data":"d005ffda8ed39bc8a828737237fdd3702ab13afec1a6ba7e1f319e7d75eed523"} Oct 31 03:19:44.395895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:44.395803 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bbvw" event={"ID":"d12ed103-14a9-4bf7-a1f6-3ccdf802838a","Type":"ContainerStarted","Data":"bab6c27217eb9eb3b0f9f73227f00e34b9f3ec89ccbab02bf9b4b084a512c907"} Oct 31 03:19:44.395895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:44.395861 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bbvw" event={"ID":"d12ed103-14a9-4bf7-a1f6-3ccdf802838a","Type":"ContainerStarted","Data":"c641f7725bae348ecc4e176cfed2d774acb86bb69999c1ffa611f6911ff35fd2"} Oct 31 03:19:44.396368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:44.395966 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:45.042173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.042112 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx"] Oct 31 03:19:45.071904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.071878 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:45.107665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.107626 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx"] Oct 31 03:19:45.157216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.157181 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pppfd\"" Oct 31 03:19:45.157383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.157285 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 03:19:45.158732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.158707 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:45.167748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.167720 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a69464e-a1dc-4a52-a80a-a53c65aafde5-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7lhsx\" (UID: \"3a69464e-a1dc-4a52-a80a-a53c65aafde5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:45.175470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.175447 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65df7c7d56-mxh75"] Oct 31 03:19:45.190415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.190397 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.221321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.221296 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65df7c7d56-mxh75"] Oct 31 03:19:45.243320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.243295 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 03:19:45.243490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.243453 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 03:19:45.243567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.243537 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8zl9c\"" Oct 31 03:19:45.268542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268510 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/427a534a-0fac-460c-bb68-1004c84aa0cb-image-registry-private-configuration\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268552 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-bound-sa-token\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268631 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a69464e-a1dc-4a52-a80a-a53c65aafde5-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7lhsx\" (UID: \"3a69464e-a1dc-4a52-a80a-a53c65aafde5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:45.268687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268660 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/427a534a-0fac-460c-bb68-1004c84aa0cb-registry-certificates\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268677 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/427a534a-0fac-460c-bb68-1004c84aa0cb-installation-pull-secrets\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268725 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/427a534a-0fac-460c-bb68-1004c84aa0cb-trusted-ca\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268822 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-kube-api-access-xzqhm\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-registry-tls\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.268893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.268883 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/427a534a-0fac-460c-bb68-1004c84aa0cb-ca-trust-extracted\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.271263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.271242 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3a69464e-a1dc-4a52-a80a-a53c65aafde5-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7lhsx\" (UID: \"3a69464e-a1dc-4a52-a80a-a53c65aafde5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:45.292852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.292781 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 03:19:45.307294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.307271 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 03:19:45.321546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.321504 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:45.369855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.369818 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/427a534a-0fac-460c-bb68-1004c84aa0cb-image-registry-private-configuration\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.369861 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-bound-sa-token\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.369890 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/427a534a-0fac-460c-bb68-1004c84aa0cb-registry-certificates\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.369993 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/427a534a-0fac-460c-bb68-1004c84aa0cb-installation-pull-secrets\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370025 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/427a534a-0fac-460c-bb68-1004c84aa0cb-trusted-ca\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370053 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-kube-api-access-xzqhm\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370122 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-registry-tls\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370148 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/427a534a-0fac-460c-bb68-1004c84aa0cb-ca-trust-extracted\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370587 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/427a534a-0fac-460c-bb68-1004c84aa0cb-ca-trust-extracted\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370756 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/427a534a-0fac-460c-bb68-1004c84aa0cb-registry-certificates\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.370908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.370892 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/427a534a-0fac-460c-bb68-1004c84aa0cb-trusted-ca\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.372861 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.372834 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-registry-tls\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.372995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.372973 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/427a534a-0fac-460c-bb68-1004c84aa0cb-installation-pull-secrets\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.373074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.373025 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/427a534a-0fac-460c-bb68-1004c84aa0cb-image-registry-private-configuration\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.402233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.402212 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:45.431245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.431220 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-bound-sa-token\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.431486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.431466 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/427a534a-0fac-460c-bb68-1004c84aa0cb-kube-api-access-xzqhm\") pod \"image-registry-65df7c7d56-mxh75\" (UID: \"427a534a-0fac-460c-bb68-1004c84aa0cb\") " pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.511136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.511104 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:45.717652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.717619 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:45.878074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.878043 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx"] Oct 31 03:19:45.884698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.884672 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:45.910030 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:45.910006 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a69464e_a1dc_4a52_a80a_a53c65aafde5.slice/crio-43f46b57a3b90be95c627bf7e8027299ce28886b5ec7804c20f99d290afdb163 WatchSource:0}: Error finding container 43f46b57a3b90be95c627bf7e8027299ce28886b5ec7804c20f99d290afdb163: Status 404 returned error can't find the container with id 43f46b57a3b90be95c627bf7e8027299ce28886b5ec7804c20f99d290afdb163 Oct 31 03:19:45.932847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:45.932816 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65df7c7d56-mxh75"] Oct 31 03:19:45.936666 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:45.936644 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427a534a_0fac_460c_bb68_1004c84aa0cb.slice/crio-18f60274a9e4dd10b4e78e267184aa6482329109bfdf8f6afe0ae19ade74539f WatchSource:0}: Error finding container 18f60274a9e4dd10b4e78e267184aa6482329109bfdf8f6afe0ae19ade74539f: Status 404 returned error can't find the container with id 18f60274a9e4dd10b4e78e267184aa6482329109bfdf8f6afe0ae19ade74539f Oct 31 03:19:46.406543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:46.406505 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" event={"ID":"3a69464e-a1dc-4a52-a80a-a53c65aafde5","Type":"ContainerStarted","Data":"43f46b57a3b90be95c627bf7e8027299ce28886b5ec7804c20f99d290afdb163"} Oct 31 03:19:46.408279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:46.408249 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" event={"ID":"427a534a-0fac-460c-bb68-1004c84aa0cb","Type":"ContainerStarted","Data":"0e432261a1a1c8a389cc1fe1ed08ac56dcce17acbc5836b997c54039d8aed285"} Oct 31 03:19:46.408425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:46.408299 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" event={"ID":"427a534a-0fac-460c-bb68-1004c84aa0cb","Type":"ContainerStarted","Data":"18f60274a9e4dd10b4e78e267184aa6482329109bfdf8f6afe0ae19ade74539f"} Oct 31 03:19:46.408425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:46.408340 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:19:46.410424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:46.410379 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qs77w" event={"ID":"9890fd99-71b5-45ab-a4f0-38293eff7802","Type":"ContainerStarted","Data":"49ec3560461d7ccbb58c53e64fcb2a34c521e0cbe572f8df60fc47f421dc8cfc"} Oct 31 03:19:46.530122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:46.530046 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" podStartSLOduration=1.5300336620000001 podStartE2EDuration="1.530033662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:46.529254225 +0000 UTC m=+14.325972483" watchObservedRunningTime="2026-10-31 03:19:46.530033662 +0000 UTC m=+14.326751916" Oct 31 03:19:47.235215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:47.235181 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:47.404904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:47.404880 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:47.414719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:47.414694 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" event={"ID":"3a69464e-a1dc-4a52-a80a-a53c65aafde5","Type":"ContainerStarted","Data":"0a907541958ab4708f0c4e82a9cb00efa70acb00af5fb635c0835a14667253db"} Oct 31 03:19:47.531579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:47.531337 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" podStartSLOduration=2.082302033 podStartE2EDuration="3.531320004s" totalImagesPullingTime="1.449017971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:47.531024966 +0000 UTC m=+15.327743225" watchObservedRunningTime="2026-10-31 03:19:47.531320004 +0000 UTC m=+15.328038263" Oct 31 03:19:48.418365 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:48.418323 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:48.422886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:48.422863 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7lhsx" Oct 31 03:19:49.133815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:49.133771 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:49.291840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:49.291805 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:49.441515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:49.441405 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:49.622804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:49.622766 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:54.401625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:54.401599 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6bbvw" Oct 31 03:19:54.888709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:54.888681 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:55.509581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:55.509532 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:55.664051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:55.664021 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.028873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.028845 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.181030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.180999 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.220746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.220719 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w"] Oct 31 03:19:56.223295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.223274 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.276048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.276018 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:19:56.276048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.276018 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:19:56.326023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.325947 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 03:19:56.326023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.325947 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 03:19:56.326023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.325975 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v78x7\"" Oct 31 03:19:56.326258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.326059 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:19:56.326796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.326778 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w"] Oct 31 03:19:56.330561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.330539 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2m89j"] Oct 31 03:19:56.332455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.332442 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.347263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.347241 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbdn\" (UniqueName: \"kubernetes.io/projected/6b2d2183-b5ee-476a-9085-4e01563d8f13-kube-api-access-4wbdn\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.347374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.347273 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b2d2183-b5ee-476a-9085-4e01563d8f13-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.347374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.347310 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b2d2183-b5ee-476a-9085-4e01563d8f13-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.347444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.347384 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2d2183-b5ee-476a-9085-4e01563d8f13-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.385150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.385115 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:19:56.385310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.385153 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.385310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.385267 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:19:56.385310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.385279 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:19:56.385461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.385341 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m85g4\"" Oct 31 03:19:56.447784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.447757 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xx6\" (UniqueName: \"kubernetes.io/projected/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-kube-api-access-n2xx6\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.447959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.447795 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b2d2183-b5ee-476a-9085-4e01563d8f13-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.447959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.447820 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-wtmp\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.447959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.447844 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448056 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2d2183-b5ee-476a-9085-4e01563d8f13-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.448172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448102 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-root\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448136 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-accelerators-collector-config\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448168 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-sys\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448208 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-tls\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448276 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbdn\" (UniqueName: \"kubernetes.io/projected/6b2d2183-b5ee-476a-9085-4e01563d8f13-kube-api-access-4wbdn\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.448376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448315 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-metrics-client-ca\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448342 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-textfile\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.448376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448372 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b2d2183-b5ee-476a-9085-4e01563d8f13-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.448767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.448750 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2d2183-b5ee-476a-9085-4e01563d8f13-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.450511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.450496 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b2d2183-b5ee-476a-9085-4e01563d8f13-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.450741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.450721 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b2d2183-b5ee-476a-9085-4e01563d8f13-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.509749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.509714 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbdn\" (UniqueName: \"kubernetes.io/projected/6b2d2183-b5ee-476a-9085-4e01563d8f13-kube-api-access-4wbdn\") pod \"openshift-state-metrics-7bd7489957-x9b6w\" (UID: \"6b2d2183-b5ee-476a-9085-4e01563d8f13\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.534558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.534529 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" Oct 31 03:19:56.537511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.537482 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.549467 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549441 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xx6\" (UniqueName: \"kubernetes.io/projected/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-kube-api-access-n2xx6\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549491 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-wtmp\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549517 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549552 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-root\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549618 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-root\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549739 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-wtmp\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549792 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-accelerators-collector-config\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549818 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-sys\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549838 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-tls\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549883 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-sys\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.549906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549878 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-metrics-client-ca\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.550113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.549925 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-textfile\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.550254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.550236 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-textfile\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.550493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.550473 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-metrics-client-ca\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.550606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.550586 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-accelerators-collector-config\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.552186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.552163 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.552263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.552252 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-node-exporter-tls\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.611055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.611027 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xx6\" (UniqueName: \"kubernetes.io/projected/02356a1a-5fe3-424d-95b3-7d3e6f7e9b06-kube-api-access-n2xx6\") pod \"node-exporter-2m89j\" (UID: \"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06\") " pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.645135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.645083 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2m89j" Oct 31 03:19:56.655015 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:56.654986 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02356a1a_5fe3_424d_95b3_7d3e6f7e9b06.slice/crio-39e718852f86779c55ba8e0eb05f688156253350bbd6bda3b71bf93acb3c259c WatchSource:0}: Error finding container 39e718852f86779c55ba8e0eb05f688156253350bbd6bda3b71bf93acb3c259c: Status 404 returned error can't find the container with id 39e718852f86779c55ba8e0eb05f688156253350bbd6bda3b71bf93acb3c259c Oct 31 03:19:56.690818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.690786 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.956035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.955936 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w"] Oct 31 03:19:56.962406 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:19:56.962379 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2d2183_b5ee_476a_9085_4e01563d8f13.slice/crio-f6537b540879bbec464d5cc468852da77f90bf7a611f94612ea2840bb845a4f7 WatchSource:0}: Error finding container f6537b540879bbec464d5cc468852da77f90bf7a611f94612ea2840bb845a4f7: Status 404 returned error can't find the container with id f6537b540879bbec464d5cc468852da77f90bf7a611f94612ea2840bb845a4f7 Oct 31 03:19:56.986027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:56.986000 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:57.448166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:57.448128 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2m89j" event={"ID":"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06","Type":"ContainerStarted","Data":"39e718852f86779c55ba8e0eb05f688156253350bbd6bda3b71bf93acb3c259c"} Oct 31 03:19:57.449965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:57.449939 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" event={"ID":"6b2d2183-b5ee-476a-9085-4e01563d8f13","Type":"ContainerStarted","Data":"756afecfc8d69d53da4332e72600231fc2f0dc45085edae78a56bcebc7b6cc0a"} Oct 31 03:19:57.450109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:57.449988 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" event={"ID":"6b2d2183-b5ee-476a-9085-4e01563d8f13","Type":"ContainerStarted","Data":"9e43fc10eb599008db1e8b4f8febd70dc1ea7f0225bba3a2fe15ba4a96aed6b6"} Oct 31 03:19:57.450109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:57.450006 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" event={"ID":"6b2d2183-b5ee-476a-9085-4e01563d8f13","Type":"ContainerStarted","Data":"f6537b540879bbec464d5cc468852da77f90bf7a611f94612ea2840bb845a4f7"} Oct 31 03:19:57.641558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:57.641516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:57.822687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:57.822601 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:58.474477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:58.474446 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:58.663806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:58.663760 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:58.820434 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:58.820340 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:59.459235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:59.459204 4488 generic.go:356] "Generic (PLEG): container finished" podID="02356a1a-5fe3-424d-95b3-7d3e6f7e9b06" containerID="d734e603b0cf592055e4894e19fc609efefbccc87e398d6e3b52788f373cae41" exitCode=0 Oct 31 03:19:59.459494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:59.459291 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2m89j" event={"ID":"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06","Type":"ContainerDied","Data":"d734e603b0cf592055e4894e19fc609efefbccc87e398d6e3b52788f373cae41"} Oct 31 03:19:59.461486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:59.461465 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" event={"ID":"6b2d2183-b5ee-476a-9085-4e01563d8f13","Type":"ContainerStarted","Data":"fff4c4e1a7bf9d9305192e797a1260d636cca242956adc90a0a6fea47d06acaa"} Oct 31 03:19:59.750149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:19:59.750046 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9b6w" podStartSLOduration=1.856916928 podStartE2EDuration="3.750030546s" totalImagesPullingTime="1.893113618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:59.748815767 +0000 UTC m=+27.545534073" watchObservedRunningTime="2026-10-31 03:19:59.750030546 +0000 UTC m=+27.546748803" Oct 31 03:20:00.316503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.316476 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:00.466609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.466546 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:00.467692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.467659 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2m89j" event={"ID":"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06","Type":"ContainerStarted","Data":"4b57d8f3751d9b5e320e48c8d89508a56927046f809aa7427a521e506c98775c"} Oct 31 03:20:00.467841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.467713 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2m89j" event={"ID":"02356a1a-5fe3-424d-95b3-7d3e6f7e9b06","Type":"ContainerStarted","Data":"cfc72a56faba7b549ea768783f53fe55f5f7bd5cfa35511a1763de5344591f40"} Oct 31 03:20:00.588391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.588335 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2m89j" podStartSLOduration=2.701164738 podStartE2EDuration="4.588319069s" totalImagesPullingTime="1.794090716s" totalInitContainerRuntime="93.063615ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:00.587942849 +0000 UTC m=+28.384661086" watchObservedRunningTime="2026-10-31 03:20:00.588319069 +0000 UTC m=+28.385037330" Oct 31 03:20:00.679450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.679417 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:00.836157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.836112 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:00.991546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:00.991464 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:01.002574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:01.002553 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4tqk" Oct 31 03:20:01.144897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:01.144864 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:02.166170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:02.166136 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:02.317971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:02.317940 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.206282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.206249 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.279621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.279589 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:05.312474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.312447 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.359771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.359742 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:05.422678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422645 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.422678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422675 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.422871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422696 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.422871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422712 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.422871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422764 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.422871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422824 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422875 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422892 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422936 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.422982 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423043 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423059 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423077 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423134 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423200 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423229 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423247 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423293 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423310 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.423361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.423335 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knm67\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.431461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.431444 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 03:20:05.431547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.431458 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 03:20:05.431547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.431444 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 03:20:05.431547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.431475 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 03:20:05.431660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.431649 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 03:20:05.431828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.431817 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 03:20:05.458077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.458022 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 03:20:05.458077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.458026 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 03:20:05.458077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.458059 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4arhh83721g0c\"" Oct 31 03:20:05.458261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.458108 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7gdr\"" Oct 31 03:20:05.458261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.458208 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 03:20:05.463945 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.463922 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:20:05.464025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.464014 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 03:20:05.464070 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.464023 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 03:20:05.464129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.464111 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 03:20:05.524653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524623 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524659 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524683 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524707 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524728 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knm67\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524752 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524771 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524802 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524829 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524855 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524881 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.524906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524912 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.525519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524933 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.525519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.524962 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.525519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.525007 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.525825 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.525795 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.526736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.526669 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.527824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.527801 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528016 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528224 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528283 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528306 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528328 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528360 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528386 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528540 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.528970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528949 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.529067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.529047 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.529408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.529000 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.529408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.529391 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.529570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.528966 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.529570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.529498 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.529831 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.529760 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.530006 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.529990 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.530779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.530760 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.531309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.531293 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.531599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.531567 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.559766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.559737 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 03:20:05.559887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.559792 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 03:20:05.567617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.567573 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.569916 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.569899 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.585867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.585843 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knm67\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67\") pod \"prometheus-k8s-0\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:05.718066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:05.717984 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:06.221008 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.220966 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:06.248955 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:20:06.248922 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7443a4_ee9d_4a75_9964_50a521aec384.slice/crio-7d21239cdda7bd53a62f74fcd9a0d3510262feb2317e1264324f9aa890a3091e WatchSource:0}: Error finding container 7d21239cdda7bd53a62f74fcd9a0d3510262feb2317e1264324f9aa890a3091e: Status 404 returned error can't find the container with id 7d21239cdda7bd53a62f74fcd9a0d3510262feb2317e1264324f9aa890a3091e Oct 31 03:20:06.322982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.322930 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:06.488814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.488735 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"7d21239cdda7bd53a62f74fcd9a0d3510262feb2317e1264324f9aa890a3091e"} Oct 31 03:20:06.489759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.489723 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:06.588708 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:06.588677 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:20:06.654769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.654732 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:06.814625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.814550 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:06.977366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:06.977330 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:07.419815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:07.419784 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65df7c7d56-mxh75" Oct 31 03:20:07.705353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:07.705259 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.003752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:08.003674 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.161454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:08.161420 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.316164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:08.316133 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.499344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:08.499312 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" exitCode=0 Oct 31 03:20:08.499771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:08.499401 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} Oct 31 03:20:08.824487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:08.824446 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:10.975549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:10.975507 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:11.125858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:11.125828 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:12.520186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:12.520148 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} Oct 31 03:20:12.520576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:12.520218 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} Oct 31 03:20:14.530800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:14.530768 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} Oct 31 03:20:14.531287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:14.530845 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} Oct 31 03:20:14.531287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:14.530891 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} Oct 31 03:20:14.531287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:14.530930 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerStarted","Data":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} Oct 31 03:20:14.811644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:14.811461 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.259511588 podStartE2EDuration="9.811443878s" totalImagesPullingTime="7.535380821s" totalInitContainerRuntime="16.551469ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:14.76107368 +0000 UTC m=+42.557791935" watchObservedRunningTime="2026-10-31 03:20:14.811443878 +0000 UTC m=+42.608162137" Oct 31 03:20:15.718599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:15.718557 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:16.203493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:16.203461 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:16.354456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:16.354425 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:16.530639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:16.530554 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:17.005137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:17.005074 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:17.153990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:17.153956 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:24.243567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:24.243524 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:24.395356 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:24.395318 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:31.256700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:31.256666 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:31.410320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:31.410289 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:32.167234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:32.167197 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:32.323755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:32.323721 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:34.290772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:34.290736 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:34.313917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:34.313894 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:34.617661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:34.617636 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:35.462033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:35.462000 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:35.630446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:35.630406 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:36.587953 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:36.587916 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:20:40.234620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:40.234585 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:40.387483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:40.387455 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:45.012826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:45.012791 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:45.166633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:45.166599 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:46.059866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:46.059833 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:46.217913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:46.217879 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:47.195364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:47.195329 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:47.371838 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:47.371806 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:48.209974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.209943 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-9vj2z-6h7j2"] Oct 31 03:20:48.210393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.210210 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" podUID="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" containerName="insights-gathering" containerID="cri-o://b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4" gracePeriod=30 Oct 31 03:20:48.454719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.454697 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:20:48.577431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577409 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") pod \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " Oct 31 03:20:48.577585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577470 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") pod \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " Oct 31 03:20:48.577585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577511 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") pod \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " Oct 31 03:20:48.577585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577540 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") pod \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\" (UID: \"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62\") " Oct 31 03:20:48.577901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577868 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle" pod "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" (UID: "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:48.578007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577894 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path" pod "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" (UID: "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:48.578007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.577955 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle" pod "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" (UID: "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:48.579856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.579836 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw" pod "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" (UID: "ef682d2d-6dfe-4b58-bf79-e7b2e268ff62"). InnerVolumeSpecName "kube-api-access-dqgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:48.646832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.646800 4488 generic.go:356] "Generic (PLEG): container finished" podID="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" containerID="b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4" exitCode=2 Oct 31 03:20:48.646987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.646855 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" Oct 31 03:20:48.646987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.646885 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" event={"ID":"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62","Type":"ContainerDied","Data":"b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4"} Oct 31 03:20:48.646987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.646937 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9vj2z-6h7j2" event={"ID":"ef682d2d-6dfe-4b58-bf79-e7b2e268ff62","Type":"ContainerDied","Data":"05be46eaa74155ca61cbef7aa49c18ac3a73c5c91477f93b5d602acd6dd13d4b"} Oct 31 03:20:48.646987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.646957 4488 scope.go:122] "RemoveContainer" containerID="b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4" Oct 31 03:20:48.655082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.655067 4488 scope.go:122] "RemoveContainer" containerID="b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4" Oct 31 03:20:48.655366 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:48.655349 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4\": container with ID starting with b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4 not found: ID does not exist" containerID="b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4" Oct 31 03:20:48.655419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.655374 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4"} err="failed to get container status \"b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4\": rpc error: code = NotFound desc = could not find container \"b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4\": container with ID starting with b5c6f1ff9cedcbc96aef7798ecd6eda116d8846f4dee5dfab22aa5dfb40641a4 not found: ID does not exist" Oct 31 03:20:48.678752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.678722 4488 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-service-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:48.678752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.678748 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-kube-api-access-dqgrw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:48.678887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.678763 4488 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-trusted-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:48.678887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.678776 4488 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62-archives-path\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:48.781700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.781667 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:48.809798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.809754 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-9vj2z-6h7j2"] Oct 31 03:20:48.819802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.819777 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-9vj2z-6h7j2"] Oct 31 03:20:48.933594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:48.933522 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:50.187638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:50.187606 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:50.314136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:50.314107 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" path="/var/lib/kubelet/pods/ef682d2d-6dfe-4b58-bf79-e7b2e268ff62/volumes" Oct 31 03:20:50.340902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:50.340873 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:50.576558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:50.576467 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.349473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349434 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:58.349930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349757 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy" containerID="cri-o://6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" gracePeriod=600 Oct 31 03:20:58.349930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349808 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-web" containerID="cri-o://3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" gracePeriod=600 Oct 31 03:20:58.349930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349736 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="prometheus" containerID="cri-o://d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" gracePeriod=600 Oct 31 03:20:58.349930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349818 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="thanos-sidecar" containerID="cri-o://10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" gracePeriod=600 Oct 31 03:20:58.349930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349819 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" gracePeriod=600 Oct 31 03:20:58.350216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.349826 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="config-reloader" containerID="cri-o://35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" gracePeriod=600 Oct 31 03:20:58.598738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.598714 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.668112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.667983 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668032 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668072 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668128 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668151 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668168 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668195 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668227 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668260 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668292 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668319 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668376 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668403 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668436 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668466 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668496 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668522 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668551 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668577 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668640 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1\") pod \"6a7443a4-ee9d-4a75-9964-50a521aec384\" (UID: \"6a7443a4-ee9d-4a75-9964-50a521aec384\") " Oct 31 03:20:58.668813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668729 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.669279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.668898 4488 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.669279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.669266 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.669857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.669398 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.669857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.669578 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.669857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.669774 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.670777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.670582 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:58.671447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.671419 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.672130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.672104 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:58.672388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.672349 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "kube-api-access-knm67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:58.672485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.672421 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.672736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.672711 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.672853 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.672802 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:58.672926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.672882 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.673364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.673336 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.673785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.673765 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.673885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.673787 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.674240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.674225 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.674369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.674351 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.674514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.674491 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.683426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683393 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config" pod "6a7443a4-ee9d-4a75-9964-50a521aec384" (UID: "6a7443a4-ee9d-4a75-9964-50a521aec384"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.683574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683552 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" exitCode=0 Oct 31 03:20:58.683625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683575 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" exitCode=0 Oct 31 03:20:58.683625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683584 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" exitCode=0 Oct 31 03:20:58.683625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683589 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" exitCode=0 Oct 31 03:20:58.683625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683594 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" exitCode=0 Oct 31 03:20:58.683625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683599 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" exitCode=0 Oct 31 03:20:58.683765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683643 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} Oct 31 03:20:58.683765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683694 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.683765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683711 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.683869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683702 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} Oct 31 03:20:58.683869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683864 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} Oct 31 03:20:58.683962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683893 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} Oct 31 03:20:58.683962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683916 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} Oct 31 03:20:58.683962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683941 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} Oct 31 03:20:58.684143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.683964 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a7443a4-ee9d-4a75-9964-50a521aec384","Type":"ContainerDied","Data":"7d21239cdda7bd53a62f74fcd9a0d3510262feb2317e1264324f9aa890a3091e"} Oct 31 03:20:58.691496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.691428 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.698855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.698835 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.707930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.707023 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.715451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.715428 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.722631 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.722612 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.730283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.730264 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.739833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.739811 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.740297 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.740261 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.740387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.740312 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} err="failed to get container status \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" Oct 31 03:20:58.740387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.740341 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.740660 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.740642 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.740748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.740668 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} err="failed to get container status \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" Oct 31 03:20:58.740748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.740685 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.740988 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.740971 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.741057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.740998 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} err="failed to get container status \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" Oct 31 03:20:58.741057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.741021 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.741437 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.741413 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.741527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.741443 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} err="failed to get container status \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" Oct 31 03:20:58.741527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.741463 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.741759 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.741733 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.741815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.741759 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} err="failed to get container status \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" Oct 31 03:20:58.741815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.741774 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.742056 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.742030 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.742331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.742060 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} err="failed to get container status \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" Oct 31 03:20:58.742331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.742078 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.742456 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:20:58.742361 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.742456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.742389 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} err="failed to get container status \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" Oct 31 03:20:58.742456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.742408 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.743213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.743185 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} err="failed to get container status \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" Oct 31 03:20:58.743213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.743212 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.743553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.743534 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} err="failed to get container status \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" Oct 31 03:20:58.743553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.743552 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.743859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.743842 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} err="failed to get container status \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" Oct 31 03:20:58.743859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.743859 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.744665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.744642 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} err="failed to get container status \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" Oct 31 03:20:58.744665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.744662 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.744994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.744962 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} err="failed to get container status \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" Oct 31 03:20:58.745122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.744995 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.745390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.745363 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} err="failed to get container status \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" Oct 31 03:20:58.745390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.745390 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.745683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.745664 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} err="failed to get container status \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" Oct 31 03:20:58.745683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.745682 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.746390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.746283 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} err="failed to get container status \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" Oct 31 03:20:58.746390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.746311 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.746614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.746593 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} err="failed to get container status \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" Oct 31 03:20:58.746614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.746612 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.747333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.747315 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} err="failed to get container status \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" Oct 31 03:20:58.747401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.747333 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.747565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.747548 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} err="failed to get container status \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" Oct 31 03:20:58.747625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.747566 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.747795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.747767 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} err="failed to get container status \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" Oct 31 03:20:58.747847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.747797 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.748101 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748047 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} err="failed to get container status \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" Oct 31 03:20:58.748212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748083 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.748381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748354 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} err="failed to get container status \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" Oct 31 03:20:58.748449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748383 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.748652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748632 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} err="failed to get container status \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" Oct 31 03:20:58.748739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748653 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.748920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748897 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} err="failed to get container status \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" Oct 31 03:20:58.748920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.748920 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.749176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749157 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} err="failed to get container status \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" Oct 31 03:20:58.749176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749173 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.749438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749424 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} err="failed to get container status \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" Oct 31 03:20:58.749438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749438 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.749687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749667 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} err="failed to get container status \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" Oct 31 03:20:58.749765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749689 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.749936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749921 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} err="failed to get container status \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" Oct 31 03:20:58.750002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.749937 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.750166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750150 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} err="failed to get container status \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" Oct 31 03:20:58.750166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750165 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.750419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750395 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} err="failed to get container status \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" Oct 31 03:20:58.750419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750417 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.750647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750630 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} err="failed to get container status \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" Oct 31 03:20:58.750718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750646 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.750883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750859 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} err="failed to get container status \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" Oct 31 03:20:58.750950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.750883 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.751253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751231 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} err="failed to get container status \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" Oct 31 03:20:58.751253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751252 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.751490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751470 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} err="failed to get container status \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" Oct 31 03:20:58.751568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751491 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.751713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751693 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} err="failed to get container status \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" Oct 31 03:20:58.751786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751714 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.751951 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751931 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} err="failed to get container status \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" Oct 31 03:20:58.752030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.751951 4488 scope.go:122] "RemoveContainer" containerID="8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221" Oct 31 03:20:58.752255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752235 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221"} err="failed to get container status \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": rpc error: code = NotFound desc = could not find container \"8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221\": container with ID starting with 8bb61c6fab21b3e916874d234438ab2522cc6ba1b1bdafeeea7f89f2d8711221 not found: ID does not exist" Oct 31 03:20:58.752336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752255 4488 scope.go:122] "RemoveContainer" containerID="6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad" Oct 31 03:20:58.752495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752472 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad"} err="failed to get container status \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": rpc error: code = NotFound desc = could not find container \"6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad\": container with ID starting with 6e177426be6988062c97292cd4775b8e4212277299a2e851705e83072185d4ad not found: ID does not exist" Oct 31 03:20:58.752565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752494 4488 scope.go:122] "RemoveContainer" containerID="3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac" Oct 31 03:20:58.752707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752690 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac"} err="failed to get container status \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": rpc error: code = NotFound desc = could not find container \"3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac\": container with ID starting with 3faf5854f41c6ae41b9fe0d4323bb7333b33fb9b176de3f0bf0f3ddd846950ac not found: ID does not exist" Oct 31 03:20:58.752773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752706 4488 scope.go:122] "RemoveContainer" containerID="10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358" Oct 31 03:20:58.752914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752899 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358"} err="failed to get container status \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": rpc error: code = NotFound desc = could not find container \"10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358\": container with ID starting with 10c81b2235bd44cf801fc63121b11ad1ef5d5c656b06f934f94e812bd2ac4358 not found: ID does not exist" Oct 31 03:20:58.752914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.752913 4488 scope.go:122] "RemoveContainer" containerID="35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c" Oct 31 03:20:58.753165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.753144 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c"} err="failed to get container status \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": rpc error: code = NotFound desc = could not find container \"35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c\": container with ID starting with 35361e7bc9532c1d8fab5014c827bc4717c86796b506f35453d94dd4b1c1557c not found: ID does not exist" Oct 31 03:20:58.753165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.753165 4488 scope.go:122] "RemoveContainer" containerID="d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459" Oct 31 03:20:58.753399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.753381 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459"} err="failed to get container status \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": rpc error: code = NotFound desc = could not find container \"d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459\": container with ID starting with d405e66b95e5ec55f96e1d6d4b97aec8aaefe79cd94ff157599cdf7ff9b7a459 not found: ID does not exist" Oct 31 03:20:58.753399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.753398 4488 scope.go:122] "RemoveContainer" containerID="04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844" Oct 31 03:20:58.753620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.753604 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844"} err="failed to get container status \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": rpc error: code = NotFound desc = could not find container \"04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844\": container with ID starting with 04d1e3cec228279d567be87f7dbc1c7add63fe3441cd2a7079fd23a5e8f2d844 not found: ID does not exist" Oct 31 03:20:58.770166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770140 4488 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770163 4488 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-metrics-client-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770176 4488 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770189 4488 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770202 4488 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770216 4488 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-kube-rbac-proxy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770229 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knm67\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-kube-api-access-knm67\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770242 4488 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-db\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770254 4488 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a7443a4-ee9d-4a75-9964-50a521aec384-tls-assets\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770267 4488 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770281 4488 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-grpc-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770293 4488 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-config\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770308 4488 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770322 4488 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a7443a4-ee9d-4a75-9964-50a521aec384-config-out\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770335 4488 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770349 4488 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-web-config\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770362 4488 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-metrics-client-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770374 4488 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7443a4-ee9d-4a75-9964-50a521aec384-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.770657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.770387 4488 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a7443a4-ee9d-4a75-9964-50a521aec384-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.846764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.846730 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:58.853482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.853455 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:58.889691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889657 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:58.889915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889902 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="config-reloader" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889919 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="config-reloader" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889934 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="thanos-sidecar" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889941 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="thanos-sidecar" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889951 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="prometheus" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889955 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="prometheus" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889960 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-web" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889965 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-web" Oct 31 03:20:58.889968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889970 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" containerName="insights-gathering" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889975 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" containerName="insights-gathering" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889987 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.889992 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890004 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-thanos" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890012 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-thanos" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890023 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="init-config-reloader" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890028 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="init-config-reloader" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890064 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890072 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-thanos" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890077 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="config-reloader" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890083 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="thanos-sidecar" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890107 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="prometheus" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890111 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" containerName="kube-rbac-proxy-web" Oct 31 03:20:58.890233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.890119 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef682d2d-6dfe-4b58-bf79-e7b2e268ff62" containerName="insights-gathering" Oct 31 03:20:58.892284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.892264 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.945873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.945787 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 03:20:58.945873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.945792 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 03:20:58.946402 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.946382 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 03:20:58.946626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.946611 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7gdr\"" Oct 31 03:20:58.946731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.946647 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 03:20:58.946777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.946766 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 03:20:58.946853 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.946839 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 03:20:58.947290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947257 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 03:20:58.947290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947263 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 03:20:58.947290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947284 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 03:20:58.947290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947290 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:20:58.948237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947385 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 03:20:58.948237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947404 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4arhh83721g0c\"" Oct 31 03:20:58.948237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947461 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 03:20:58.948237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.947564 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 03:20:58.971553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971521 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971558 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971575 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971598 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-config\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971670 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971742 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971789 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.971903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971815 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971898 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971934 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.971961 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972059 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972101 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fql\" (UniqueName: \"kubernetes.io/projected/3c3afbd4-990d-4870-87b5-f05c511517c0-kube-api-access-r2fql\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972134 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972180 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972218 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c3afbd4-990d-4870-87b5-f05c511517c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972258 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972336 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c3afbd4-990d-4870-87b5-f05c511517c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972363 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.972479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.972390 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:58.996510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.996421 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 03:20:58.997639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.997611 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 03:20:58.998558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:58.998539 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:59.072835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072800 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-config\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.072835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072841 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072867 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072891 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072914 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072947 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072972 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.072991 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073024 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073049 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fql\" (UniqueName: \"kubernetes.io/projected/3c3afbd4-990d-4870-87b5-f05c511517c0-kube-api-access-r2fql\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073078 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073126 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073148 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c3afbd4-990d-4870-87b5-f05c511517c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073169 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073216 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c3afbd4-990d-4870-87b5-f05c511517c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073240 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073271 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073296 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073338 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.073746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073364 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.074183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073887 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.074183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.073934 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.074183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.074039 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.074919 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.074385 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.074919 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.074577 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.076363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.075223 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.076705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.076680 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-config\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.076798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.076744 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.077059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.077015 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c3afbd4-990d-4870-87b5-f05c511517c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.077283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.077259 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.077402 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.077363 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.078168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.077798 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.078168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.078126 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.078329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.078233 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c3afbd4-990d-4870-87b5-f05c511517c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.078499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.078479 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.078560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.078529 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c3afbd4-990d-4870-87b5-f05c511517c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.079329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.079310 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.079418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.079348 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.079566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.079552 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c3afbd4-990d-4870-87b5-f05c511517c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.157020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.156989 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fql\" (UniqueName: \"kubernetes.io/projected/3c3afbd4-990d-4870-87b5-f05c511517c0-kube-api-access-r2fql\") pod \"prometheus-k8s-0\" (UID: \"3c3afbd4-990d-4870-87b5-f05c511517c0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.210112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.210015 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:20:59.654344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.654313 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:20:59.658668 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:20:59.658642 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3afbd4_990d_4870_87b5_f05c511517c0.slice/crio-06c2020443ce64f62789d10cf0fa5d0535c99760794850019fcb98d84429b37d WatchSource:0}: Error finding container 06c2020443ce64f62789d10cf0fa5d0535c99760794850019fcb98d84429b37d: Status 404 returned error can't find the container with id 06c2020443ce64f62789d10cf0fa5d0535c99760794850019fcb98d84429b37d Oct 31 03:20:59.687834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:20:59.687799 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"06c2020443ce64f62789d10cf0fa5d0535c99760794850019fcb98d84429b37d"} Oct 31 03:21:00.314010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:00.313975 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7443a4-ee9d-4a75-9964-50a521aec384" path="/var/lib/kubelet/pods/6a7443a4-ee9d-4a75-9964-50a521aec384/volumes" Oct 31 03:21:00.693680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:00.693644 4488 generic.go:356] "Generic (PLEG): container finished" podID="3c3afbd4-990d-4870-87b5-f05c511517c0" containerID="cae2b3ddad881b5bab5cd1f81507365ed1d7e06407b94232b1630bef4c35fd6e" exitCode=0 Oct 31 03:21:00.694212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:00.693742 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerDied","Data":"cae2b3ddad881b5bab5cd1f81507365ed1d7e06407b94232b1630bef4c35fd6e"} Oct 31 03:21:01.700820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.700780 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"585305bf703dca8ff0af0f3882b99c9e105c271a9a9acb982d4d8a4597e60419"} Oct 31 03:21:01.701268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.700841 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"7947e554ecb907cdaf70dd5d47cac6b33c5576e4b2b5408edf6e8ad1d9981a6f"} Oct 31 03:21:01.701268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.700866 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"d9c67250c79aaf9fec49e1987beafc1c0be9318571c0490aa0cc52d50b4f055b"} Oct 31 03:21:01.701268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.700894 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"385c94126bb69c4ebb819f6d05d7a9c8a34edea7d59d3fa5a47ce2ad85d3ecef"} Oct 31 03:21:01.701268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.700916 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"8a1c65c87f8344cfc547941d649cdbc636639aca2fca08b08ea0762a4d34115f"} Oct 31 03:21:01.701268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.700937 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c3afbd4-990d-4870-87b5-f05c511517c0","Type":"ContainerStarted","Data":"949dadf7b0ed13020dd0e1fc9dfa70f8fb7ba1349d2209cbf39d4126906da961"} Oct 31 03:21:01.834890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:01.834835 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.813038665 podStartE2EDuration="3.834821764s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.783099ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:01.833788511 +0000 UTC m=+89.630506811" watchObservedRunningTime="2026-10-31 03:21:01.834821764 +0000 UTC m=+89.631540021" Oct 31 03:21:02.172135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:02.172081 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:02.338382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:02.338349 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:04.210993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:04.210951 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:21:06.588150 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:21:06.588118 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:21:16.690586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:16.690547 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:36.587940 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:21:36.587905 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:21:59.210767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:59.210729 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:21:59.227063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:59.227034 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:21:59.915507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:21:59.915483 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:22:06.586765 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:22:06.586724 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:22:17.055407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:22:17.055346 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:17.296189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:22:17.296131 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:26.745617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:22:26.745539 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:36.586984 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:22:36.586945 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:23:06.589503 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:23:06.589462 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:23:08.056660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:08.056617 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:08.215362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:08.215327 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:17.387513 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:17.387415 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:18.207118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:18.207052 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:18.379041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:18.379005 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:18.549393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:18.549313 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:19.789001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:19.788969 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:19.944907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:19.944880 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:21.155619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:21.155591 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:25.451275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:25.451244 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:25.629329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:25.629297 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:25.844538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:25.844506 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:26.853492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:26.853454 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:27.015779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:27.015682 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:27.181273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:27.181223 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:27.340346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:27.340311 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:30.515623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:30.515590 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:36.587815 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:23:36.587776 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:23:49.033574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:49.033539 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:49.208291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:49.208257 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:49.914411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:49.914371 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:50.164223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.164187 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w"] Oct 31 03:23:50.167227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.167184 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.194695 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.194663 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.194857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.194709 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.194857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.194742 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmhn\" (UniqueName: \"kubernetes.io/projected/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-kube-api-access-htmhn\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.194857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.194812 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.194857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.194841 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.194986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.194872 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.201065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.201031 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:50.241010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.240977 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:23:50.241010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.240999 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:23:50.241264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.241071 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:23:50.270791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.270754 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:23:50.270966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.270903 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:23:50.270966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.270906 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jccsh\"" Oct 31 03:23:50.272378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.272357 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w"] Oct 31 03:23:50.295767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.295735 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.295941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.295776 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmhn\" (UniqueName: \"kubernetes.io/projected/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-kube-api-access-htmhn\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.295941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.295803 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.295941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.295830 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.295941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.295870 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.295941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.295915 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.296400 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.296374 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.298489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.298467 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.298592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.298515 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.298708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.298691 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.298809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.298794 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.361386 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.361350 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmhn\" (UniqueName: \"kubernetes.io/projected/e1a556e0-8e51-4a2e-b328-4ca9af7e37c8-kube-api-access-htmhn\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-6zj7w\" (UID: \"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.389790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.389753 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:50.483460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.483375 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:50.552409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.552382 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:50.985380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:50.985357 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w"] Oct 31 03:23:50.987532 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:23:50.987510 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a556e0_8e51_4a2e_b328_4ca9af7e37c8.slice/crio-7e05651ec8d86fdb8eff8fcec6fa2833151b38a98a722fb62723a228931ef7e0 WatchSource:0}: Error finding container 7e05651ec8d86fdb8eff8fcec6fa2833151b38a98a722fb62723a228931ef7e0: Status 404 returned error can't find the container with id 7e05651ec8d86fdb8eff8fcec6fa2833151b38a98a722fb62723a228931ef7e0 Oct 31 03:23:51.291913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:51.291833 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" event={"ID":"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8","Type":"ContainerStarted","Data":"7e05651ec8d86fdb8eff8fcec6fa2833151b38a98a722fb62723a228931ef7e0"} Oct 31 03:23:51.342591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:51.342562 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:52.846451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:52.846414 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:52.999556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:52.999519 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:55.220920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:55.220882 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:55.308671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:55.308636 4488 generic.go:356] "Generic (PLEG): container finished" podID="e1a556e0-8e51-4a2e-b328-4ca9af7e37c8" containerID="70497850f643f3eff3e29eed850c50b5bd85ea9342d51013fbd8e3d5fdbacea1" exitCode=0 Oct 31 03:23:55.308857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:55.308690 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" event={"ID":"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8","Type":"ContainerDied","Data":"70497850f643f3eff3e29eed850c50b5bd85ea9342d51013fbd8e3d5fdbacea1"} Oct 31 03:23:56.314829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:56.314793 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" event={"ID":"e1a556e0-8e51-4a2e-b328-4ca9af7e37c8","Type":"ContainerStarted","Data":"3afbaf0c72a0a8d286f56940674a32831a7cc7d9c7e1746df684742db782d286"} Oct 31 03:23:56.315282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:56.314927 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:23:56.588982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:56.588926 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" podStartSLOduration=1.9582586370000001 podStartE2EDuration="6.588912708s" totalImagesPullingTime="4.203437738s" totalInitContainerRuntime="427.216333ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:56.588272122 +0000 UTC m=+264.384990382" watchObservedRunningTime="2026-10-31 03:23:56.588912708 +0000 UTC m=+264.385630964" Oct 31 03:23:59.052548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:59.052506 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:59.256508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:23:59.256477 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:01.593570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:01.593523 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:01.759519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:01.759483 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:02.124455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:02.124408 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:02.809586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:02.809550 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:03.732382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:03.732347 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:03.908021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:03.907966 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:05.584271 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:05.584239 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:05.738961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:05.738931 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:06.588047 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:24:06.588015 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:24:06.823128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:06.823068 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:06.995267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:06.995194 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf"] Oct 31 03:24:07.002329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.002300 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.105873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.105833 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hpfq7\"" Oct 31 03:24:07.105873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.105879 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 03:24:07.106211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.105881 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 03:24:07.106948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.106924 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf"] Oct 31 03:24:07.128319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.128293 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af3162c5-fd5c-49bc-97f6-0897c174cc11-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kw8rf\" (UID: \"af3162c5-fd5c-49bc-97f6-0897c174cc11\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.128476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.128333 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxb4\" (UniqueName: \"kubernetes.io/projected/af3162c5-fd5c-49bc-97f6-0897c174cc11-kube-api-access-5gxb4\") pod \"cert-manager-webhook-5dbbbf75cc-kw8rf\" (UID: \"af3162c5-fd5c-49bc-97f6-0897c174cc11\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.229332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.229290 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af3162c5-fd5c-49bc-97f6-0897c174cc11-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kw8rf\" (UID: \"af3162c5-fd5c-49bc-97f6-0897c174cc11\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.229538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.229345 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxb4\" (UniqueName: \"kubernetes.io/projected/af3162c5-fd5c-49bc-97f6-0897c174cc11-kube-api-access-5gxb4\") pod \"cert-manager-webhook-5dbbbf75cc-kw8rf\" (UID: \"af3162c5-fd5c-49bc-97f6-0897c174cc11\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.295191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.295123 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxb4\" (UniqueName: \"kubernetes.io/projected/af3162c5-fd5c-49bc-97f6-0897c174cc11-kube-api-access-5gxb4\") pod \"cert-manager-webhook-5dbbbf75cc-kw8rf\" (UID: \"af3162c5-fd5c-49bc-97f6-0897c174cc11\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.295191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.295126 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af3162c5-fd5c-49bc-97f6-0897c174cc11-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kw8rf\" (UID: \"af3162c5-fd5c-49bc-97f6-0897c174cc11\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.321549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.321521 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-6zj7w" Oct 31 03:24:07.322229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.322209 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:07.451714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.451681 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:07.615205 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.615168 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:07.697483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:07.697454 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf"] Oct 31 03:24:07.701046 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:07.701003 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf3162c5_fd5c_49bc_97f6_0897c174cc11.slice/crio-325ef7cb65be10568cb73b164b39d970a52ddcd669a4d4ee9d28d79edb9246c7 WatchSource:0}: Error finding container 325ef7cb65be10568cb73b164b39d970a52ddcd669a4d4ee9d28d79edb9246c7: Status 404 returned error can't find the container with id 325ef7cb65be10568cb73b164b39d970a52ddcd669a4d4ee9d28d79edb9246c7 Oct 31 03:24:08.012528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.012443 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:08.355497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.355463 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" event={"ID":"af3162c5-fd5c-49bc-97f6-0897c174cc11","Type":"ContainerStarted","Data":"325ef7cb65be10568cb73b164b39d970a52ddcd669a4d4ee9d28d79edb9246c7"} Oct 31 03:24:08.503644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.503608 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:08.673245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.673154 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg"] Oct 31 03:24:08.675723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.675708 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.731629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.731599 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-q47ck\"" Oct 31 03:24:08.735569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.735533 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg"] Oct 31 03:24:08.743679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.743643 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtqz\" (UniqueName: \"kubernetes.io/projected/23efadbf-bd80-415d-94c1-37d149b80ca6-kube-api-access-cgtqz\") pod \"cert-manager-cainjector-5c57bbd4fc-8xsxg\" (UID: \"23efadbf-bd80-415d-94c1-37d149b80ca6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.743837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.743684 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23efadbf-bd80-415d-94c1-37d149b80ca6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8xsxg\" (UID: \"23efadbf-bd80-415d-94c1-37d149b80ca6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.844532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.844487 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtqz\" (UniqueName: \"kubernetes.io/projected/23efadbf-bd80-415d-94c1-37d149b80ca6-kube-api-access-cgtqz\") pod \"cert-manager-cainjector-5c57bbd4fc-8xsxg\" (UID: \"23efadbf-bd80-415d-94c1-37d149b80ca6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.844532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.844528 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23efadbf-bd80-415d-94c1-37d149b80ca6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8xsxg\" (UID: \"23efadbf-bd80-415d-94c1-37d149b80ca6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.906196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.906167 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23efadbf-bd80-415d-94c1-37d149b80ca6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8xsxg\" (UID: \"23efadbf-bd80-415d-94c1-37d149b80ca6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.906352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.906220 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtqz\" (UniqueName: \"kubernetes.io/projected/23efadbf-bd80-415d-94c1-37d149b80ca6-kube-api-access-cgtqz\") pod \"cert-manager-cainjector-5c57bbd4fc-8xsxg\" (UID: \"23efadbf-bd80-415d-94c1-37d149b80ca6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:08.985759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:08.985675 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" Oct 31 03:24:09.372680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:09.372646 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg"] Oct 31 03:24:09.379153 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:09.379123 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23efadbf_bd80_415d_94c1_37d149b80ca6.slice/crio-2f01d020e366914cadf080803e3eeda136518dd4c17b5b535840e72b720ccd2a WatchSource:0}: Error finding container 2f01d020e366914cadf080803e3eeda136518dd4c17b5b535840e72b720ccd2a: Status 404 returned error can't find the container with id 2f01d020e366914cadf080803e3eeda136518dd4c17b5b535840e72b720ccd2a Oct 31 03:24:10.365335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:10.365297 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" event={"ID":"23efadbf-bd80-415d-94c1-37d149b80ca6","Type":"ContainerStarted","Data":"2f01d020e366914cadf080803e3eeda136518dd4c17b5b535840e72b720ccd2a"} Oct 31 03:24:11.841993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:11.841955 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:12.003326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:12.003293 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:13.384430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:13.384390 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" event={"ID":"af3162c5-fd5c-49bc-97f6-0897c174cc11","Type":"ContainerStarted","Data":"e97a23f07f5f592647ef01d4b6f2f79c89fa4b20e48b9bc978edd71b1dcf20a2"} Oct 31 03:24:13.384949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:13.384449 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:13.386031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:13.386007 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" event={"ID":"23efadbf-bd80-415d-94c1-37d149b80ca6","Type":"ContainerStarted","Data":"45ffb764c9c42930b10bdc1938a49efd12c55cf741c514cfa13ae2c6227c9e6b"} Oct 31 03:24:13.504632 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:13.504572 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" podStartSLOduration=2.8247873820000002 podStartE2EDuration="7.504556173s" totalImagesPullingTime="4.679768791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:13.503776743 +0000 UTC m=+281.300495001" watchObservedRunningTime="2026-10-31 03:24:13.504556173 +0000 UTC m=+281.301274433" Oct 31 03:24:13.650240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:13.650083 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8xsxg" podStartSLOduration=2.6602819699999998 podStartE2EDuration="5.650065374s" totalImagesPullingTime="2.989783404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:13.649602035 +0000 UTC m=+281.446320293" watchObservedRunningTime="2026-10-31 03:24:13.650065374 +0000 UTC m=+281.446783632" Oct 31 03:24:13.972119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:13.971986 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:14.621736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.621704 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj"] Oct 31 03:24:14.624620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.624604 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.685391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.685361 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-xjwx8\"" Oct 31 03:24:14.694022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.693989 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbc8\" (UniqueName: \"kubernetes.io/projected/40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb-kube-api-access-jhbc8\") pod \"jobset-operator-674cd59dc5-bnwgj\" (UID: \"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.694115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.694048 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb-tmp\") pod \"jobset-operator-674cd59dc5-bnwgj\" (UID: \"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.735749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.735709 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:24:14.735923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.735758 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 03:24:14.736757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.736737 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj"] Oct 31 03:24:14.794884 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.794852 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbc8\" (UniqueName: \"kubernetes.io/projected/40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb-kube-api-access-jhbc8\") pod \"jobset-operator-674cd59dc5-bnwgj\" (UID: \"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.795042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.794918 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb-tmp\") pod \"jobset-operator-674cd59dc5-bnwgj\" (UID: \"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.795401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.795382 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb-tmp\") pod \"jobset-operator-674cd59dc5-bnwgj\" (UID: \"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.858670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.858637 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbc8\" (UniqueName: \"kubernetes.io/projected/40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb-kube-api-access-jhbc8\") pod \"jobset-operator-674cd59dc5-bnwgj\" (UID: \"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:14.937124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:14.937012 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" Oct 31 03:24:15.070336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:15.070305 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.314411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:15.314339 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj"] Oct 31 03:24:15.317678 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:15.317651 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40dd4c02_5e99_4e32_9d68_24ed5f1cb7bb.slice/crio-e0eed7b9c73fa233c659819b48f6d225de5ef84c81a5c798f88a5ef6fcf47e71 WatchSource:0}: Error finding container e0eed7b9c73fa233c659819b48f6d225de5ef84c81a5c798f88a5ef6fcf47e71: Status 404 returned error can't find the container with id e0eed7b9c73fa233c659819b48f6d225de5ef84c81a5c798f88a5ef6fcf47e71 Oct 31 03:24:15.395027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:15.394993 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" event={"ID":"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb","Type":"ContainerStarted","Data":"e0eed7b9c73fa233c659819b48f6d225de5ef84c81a5c798f88a5ef6fcf47e71"} Oct 31 03:24:16.235157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:16.235121 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:16.443868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:16.443838 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:17.424756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.424723 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-wscqb"] Oct 31 03:24:17.428699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.428682 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.518428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.518384 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfebb280-0d8d-4ac8-ae76-439902538cbf-bound-sa-token\") pod \"cert-manager-597b95cb84-wscqb\" (UID: \"bfebb280-0d8d-4ac8-ae76-439902538cbf\") " pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.518428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.518434 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmlqp\" (UniqueName: \"kubernetes.io/projected/bfebb280-0d8d-4ac8-ae76-439902538cbf-kube-api-access-gmlqp\") pod \"cert-manager-597b95cb84-wscqb\" (UID: \"bfebb280-0d8d-4ac8-ae76-439902538cbf\") " pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.531350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.531326 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6sf5m\"" Oct 31 03:24:17.532344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.532324 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wscqb"] Oct 31 03:24:17.619468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.619436 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmlqp\" (UniqueName: \"kubernetes.io/projected/bfebb280-0d8d-4ac8-ae76-439902538cbf-kube-api-access-gmlqp\") pod \"cert-manager-597b95cb84-wscqb\" (UID: \"bfebb280-0d8d-4ac8-ae76-439902538cbf\") " pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.619653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.619519 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfebb280-0d8d-4ac8-ae76-439902538cbf-bound-sa-token\") pod \"cert-manager-597b95cb84-wscqb\" (UID: \"bfebb280-0d8d-4ac8-ae76-439902538cbf\") " pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.688502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.688425 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfebb280-0d8d-4ac8-ae76-439902538cbf-bound-sa-token\") pod \"cert-manager-597b95cb84-wscqb\" (UID: \"bfebb280-0d8d-4ac8-ae76-439902538cbf\") " pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.688502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.688476 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmlqp\" (UniqueName: \"kubernetes.io/projected/bfebb280-0d8d-4ac8-ae76-439902538cbf-kube-api-access-gmlqp\") pod \"cert-manager-597b95cb84-wscqb\" (UID: \"bfebb280-0d8d-4ac8-ae76-439902538cbf\") " pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:17.730602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.730563 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:17.740577 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:17.740545 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wscqb" Oct 31 03:24:18.123192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:18.123160 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wscqb"] Oct 31 03:24:18.126153 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:18.126128 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfebb280_0d8d_4ac8_ae76_439902538cbf.slice/crio-04e58aa968b3065f8f69f6c52391b2a52b5d74ba14218a0a5898345b2940d434 WatchSource:0}: Error finding container 04e58aa968b3065f8f69f6c52391b2a52b5d74ba14218a0a5898345b2940d434: Status 404 returned error can't find the container with id 04e58aa968b3065f8f69f6c52391b2a52b5d74ba14218a0a5898345b2940d434 Oct 31 03:24:18.412606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:18.412510 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wscqb" event={"ID":"bfebb280-0d8d-4ac8-ae76-439902538cbf","Type":"ContainerStarted","Data":"d7320458f01262a14e641ec6c7f42d0218b8751bc1aba4a71347f856d1822c36"} Oct 31 03:24:18.412606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:18.412560 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wscqb" event={"ID":"bfebb280-0d8d-4ac8-ae76-439902538cbf","Type":"ContainerStarted","Data":"04e58aa968b3065f8f69f6c52391b2a52b5d74ba14218a0a5898345b2940d434"} Oct 31 03:24:18.414968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:18.414945 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:18.552846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:18.552774 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-wscqb" podStartSLOduration=1.5527564680000001 podStartE2EDuration="1.552756468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:18.55216456 +0000 UTC m=+286.348882817" watchObservedRunningTime="2026-10-31 03:24:18.552756468 +0000 UTC m=+286.349474728" Oct 31 03:24:18.868157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:18.868121 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:19.034037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:19.033995 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:19.392881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:19.392850 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" Oct 31 03:24:20.298278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:20.298237 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:20.511743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:20.511709 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:21.286374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:21.286339 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:21.431199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:21.431159 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" event={"ID":"40dd4c02-5e99-4e32-9d68-24ed5f1cb7bb","Type":"ContainerStarted","Data":"ac4a70f06776ad0ad238d988ded1580e6e5aa9de2362ba9b74a2c2b93c8234f4"} Oct 31 03:24:21.436373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:21.436346 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:21.587398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:21.587341 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bnwgj" podStartSLOduration=2.377276832 podStartE2EDuration="7.587326254s" totalImagesPullingTime="5.210049422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:21.58664693 +0000 UTC m=+289.383365189" watchObservedRunningTime="2026-10-31 03:24:21.587326254 +0000 UTC m=+289.384044510" Oct 31 03:24:22.705134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:22.705076 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.870610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:22.870577 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:23.390135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.390079 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-764498d796-5l9zv"] Oct 31 03:24:23.393275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.393245 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:23.393691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.393673 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.470353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.470320 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcb6\" (UniqueName: \"kubernetes.io/projected/f4ac29c7-2389-44ce-90af-a241e3e98271-kube-api-access-mdcb6\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.470537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.470365 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ac29c7-2389-44ce-90af-a241e3e98271-tls-certs\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.470537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.470445 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4ac29c7-2389-44ce-90af-a241e3e98271-tmp\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.497187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.497146 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cxr2k\"" Oct 31 03:24:23.497373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.497264 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:24:23.497373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.497364 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:24:23.497489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.497406 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:24:23.498544 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.498524 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-764498d796-5l9zv"] Oct 31 03:24:23.548161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.548129 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 03:24:23.567940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.567912 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:23.571010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.570990 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcb6\" (UniqueName: \"kubernetes.io/projected/f4ac29c7-2389-44ce-90af-a241e3e98271-kube-api-access-mdcb6\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.571166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.571036 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ac29c7-2389-44ce-90af-a241e3e98271-tls-certs\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.571166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.571142 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4ac29c7-2389-44ce-90af-a241e3e98271-tmp\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.573686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.573657 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4ac29c7-2389-44ce-90af-a241e3e98271-tmp\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.573804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.573708 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ac29c7-2389-44ce-90af-a241e3e98271-tls-certs\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.633588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.633560 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcb6\" (UniqueName: \"kubernetes.io/projected/f4ac29c7-2389-44ce-90af-a241e3e98271-kube-api-access-mdcb6\") pod \"kube-auth-proxy-764498d796-5l9zv\" (UID: \"f4ac29c7-2389-44ce-90af-a241e3e98271\") " pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:23.705654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:23.705575 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" Oct 31 03:24:24.030014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:24.029927 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:24.084183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:24.084150 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-764498d796-5l9zv"] Oct 31 03:24:24.087390 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:24.087353 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ac29c7_2389_44ce_90af_a241e3e98271.slice/crio-712a03e2e31ec5cde0f811aad2af2111723e364faff69cdfebe9bf4fdefae78c WatchSource:0}: Error finding container 712a03e2e31ec5cde0f811aad2af2111723e364faff69cdfebe9bf4fdefae78c: Status 404 returned error can't find the container with id 712a03e2e31ec5cde0f811aad2af2111723e364faff69cdfebe9bf4fdefae78c Oct 31 03:24:24.200723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:24.200691 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:24.353369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:24.353335 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:24.445956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:24.445920 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" event={"ID":"f4ac29c7-2389-44ce-90af-a241e3e98271","Type":"ContainerStarted","Data":"712a03e2e31ec5cde0f811aad2af2111723e364faff69cdfebe9bf4fdefae78c"} Oct 31 03:24:24.870631 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:24.870590 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.818514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:26.818423 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.980299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:26.980256 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:27.136711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:27.136669 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:28.032748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.032712 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gv22m"] Oct 31 03:24:28.036020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.035987 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:28.088471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.088430 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:24:28.089029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.089013 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:24:28.119330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.119291 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbcm\" (UniqueName: \"kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm\") pod \"authorino-57f7f8659f-gv22m\" (UID: \"0ad1edce-bfe0-4d59-a377-bb9818d7ec71\") " pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:28.138554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.138516 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bq6cs\"" Oct 31 03:24:28.139845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.139821 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gv22m"] Oct 31 03:24:28.192604 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.192536 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:28.220728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.220687 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbcm\" (UniqueName: \"kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm\") pod \"authorino-57f7f8659f-gv22m\" (UID: \"0ad1edce-bfe0-4d59-a377-bb9818d7ec71\") " pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:28.282283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.282256 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbcm\" (UniqueName: \"kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm\") pod \"authorino-57f7f8659f-gv22m\" (UID: \"0ad1edce-bfe0-4d59-a377-bb9818d7ec71\") " pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:28.347619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.347585 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:28.351493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.351417 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:28.467281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.467246 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" event={"ID":"f4ac29c7-2389-44ce-90af-a241e3e98271","Type":"ContainerStarted","Data":"e22f4d4b8a6f4499984c8a5437736e8928aa24aae3915f53ff1e9636198691fc"} Oct 31 03:24:28.585548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.585489 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-764498d796-5l9zv" podStartSLOduration=2.11463446 podStartE2EDuration="5.585473379s" totalImagesPullingTime="3.470838919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:28.584058323 +0000 UTC m=+296.380776583" watchObservedRunningTime="2026-10-31 03:24:28.585473379 +0000 UTC m=+296.382191635" Oct 31 03:24:28.717886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:28.717811 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gv22m"] Oct 31 03:24:28.721165 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:28.721135 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ad1edce_bfe0_4d59_a377_bb9818d7ec71.slice/crio-32b6ce539e3f8bd1c09d8ba1d7b3ceb22980b0e134de8633239d61200dcc321f WatchSource:0}: Error finding container 32b6ce539e3f8bd1c09d8ba1d7b3ceb22980b0e134de8633239d61200dcc321f: Status 404 returned error can't find the container with id 32b6ce539e3f8bd1c09d8ba1d7b3ceb22980b0e134de8633239d61200dcc321f Oct 31 03:24:29.218065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.218031 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:29.369343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.369315 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:29.472738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.472645 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gv22m" event={"ID":"0ad1edce-bfe0-4d59-a377-bb9818d7ec71","Type":"ContainerStarted","Data":"32b6ce539e3f8bd1c09d8ba1d7b3ceb22980b0e134de8633239d61200dcc321f"} Oct 31 03:24:29.614970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.614933 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:29.771551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.771465 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:29.830858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.830826 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:24:29.835329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.835307 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:29.887814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.887777 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mggw\"" Oct 31 03:24:29.887814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.887811 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 03:24:29.892748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.892722 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:24:29.936922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.936888 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ef7c187-78c1-47cd-86e6-4c4af998f3ba-config-file\") pod \"limitador-limitador-7c9b87d574-gc8k4\" (UID: \"9ef7c187-78c1-47cd-86e6-4c4af998f3ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:29.937140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.936938 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bdn\" (UniqueName: \"kubernetes.io/projected/9ef7c187-78c1-47cd-86e6-4c4af998f3ba-kube-api-access-27bdn\") pod \"limitador-limitador-7c9b87d574-gc8k4\" (UID: \"9ef7c187-78c1-47cd-86e6-4c4af998f3ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:29.975653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:29.975621 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.038482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.038399 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ef7c187-78c1-47cd-86e6-4c4af998f3ba-config-file\") pod \"limitador-limitador-7c9b87d574-gc8k4\" (UID: \"9ef7c187-78c1-47cd-86e6-4c4af998f3ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:30.038482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.038442 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bdn\" (UniqueName: \"kubernetes.io/projected/9ef7c187-78c1-47cd-86e6-4c4af998f3ba-kube-api-access-27bdn\") pod \"limitador-limitador-7c9b87d574-gc8k4\" (UID: \"9ef7c187-78c1-47cd-86e6-4c4af998f3ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:30.039050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.039030 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ef7c187-78c1-47cd-86e6-4c4af998f3ba-config-file\") pod \"limitador-limitador-7c9b87d574-gc8k4\" (UID: \"9ef7c187-78c1-47cd-86e6-4c4af998f3ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:30.099833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.099798 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bdn\" (UniqueName: \"kubernetes.io/projected/9ef7c187-78c1-47cd-86e6-4c4af998f3ba-kube-api-access-27bdn\") pod \"limitador-limitador-7c9b87d574-gc8k4\" (UID: \"9ef7c187-78c1-47cd-86e6-4c4af998f3ba\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:30.139548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.139511 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.148491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.148464 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:30.155068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.155042 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:24:30.290757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.290649 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.521595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.521564 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:24:30.524237 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:30.524211 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef7c187_78c1_47cd_86e6_4c4af998f3ba.slice/crio-bf469993e5d2544455bfb605b14b430d622875639d4cffcd000b27fc5f5fdb0a WatchSource:0}: Error finding container bf469993e5d2544455bfb605b14b430d622875639d4cffcd000b27fc5f5fdb0a: Status 404 returned error can't find the container with id bf469993e5d2544455bfb605b14b430d622875639d4cffcd000b27fc5f5fdb0a Oct 31 03:24:30.748995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:30.748954 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:31.481937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:31.481896 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" event={"ID":"9ef7c187-78c1-47cd-86e6-4c4af998f3ba","Type":"ContainerStarted","Data":"bf469993e5d2544455bfb605b14b430d622875639d4cffcd000b27fc5f5fdb0a"} Oct 31 03:24:33.324823 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:33.324783 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gv22m"] Oct 31 03:24:33.492606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:33.492565 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" event={"ID":"9ef7c187-78c1-47cd-86e6-4c4af998f3ba","Type":"ContainerStarted","Data":"60d231a6cda9ab49f13742d6731ec2afc5b6c74142ff690cb58101933370cb8a"} Oct 31 03:24:33.492804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:33.492778 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:33.494058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:33.494036 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gv22m" event={"ID":"0ad1edce-bfe0-4d59-a377-bb9818d7ec71","Type":"ContainerStarted","Data":"b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5"} Oct 31 03:24:33.609674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:33.609620 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" podStartSLOduration=2.366030939 podStartE2EDuration="4.609605702s" totalImagesPullingTime="2.243574763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:33.609055634 +0000 UTC m=+301.405773895" watchObservedRunningTime="2026-10-31 03:24:33.609605702 +0000 UTC m=+301.406323959" Oct 31 03:24:33.776699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:33.776638 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-gv22m" podStartSLOduration=2.805588869 podStartE2EDuration="6.776621245s" totalImagesPullingTime="3.971032376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:33.776031351 +0000 UTC m=+301.572749606" watchObservedRunningTime="2026-10-31 03:24:33.776621245 +0000 UTC m=+301.573339505" Oct 31 03:24:34.497909 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:34.497849 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-gv22m" podUID="0ad1edce-bfe0-4d59-a377-bb9818d7ec71" containerName="authorino" containerID="cri-o://b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5" gracePeriod=30 Oct 31 03:24:34.767880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:34.767852 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:34.882164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:34.882123 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm\" (UniqueName: \"kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm\") pod \"0ad1edce-bfe0-4d59-a377-bb9818d7ec71\" (UID: \"0ad1edce-bfe0-4d59-a377-bb9818d7ec71\") " Oct 31 03:24:34.884606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:34.884572 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm" pod "0ad1edce-bfe0-4d59-a377-bb9818d7ec71" (UID: "0ad1edce-bfe0-4d59-a377-bb9818d7ec71"). InnerVolumeSpecName "kube-api-access-bwbcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:34.983265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:34.983224 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwbcm\" (UniqueName: \"kubernetes.io/projected/0ad1edce-bfe0-4d59-a377-bb9818d7ec71-kube-api-access-bwbcm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:24:35.503031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.503000 4488 generic.go:356] "Generic (PLEG): container finished" podID="0ad1edce-bfe0-4d59-a377-bb9818d7ec71" containerID="b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5" exitCode=0 Oct 31 03:24:35.503498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.503047 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gv22m" Oct 31 03:24:35.503498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.503118 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gv22m" event={"ID":"0ad1edce-bfe0-4d59-a377-bb9818d7ec71","Type":"ContainerDied","Data":"b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5"} Oct 31 03:24:35.503498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.503167 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gv22m" event={"ID":"0ad1edce-bfe0-4d59-a377-bb9818d7ec71","Type":"ContainerDied","Data":"32b6ce539e3f8bd1c09d8ba1d7b3ceb22980b0e134de8633239d61200dcc321f"} Oct 31 03:24:35.503498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.503191 4488 scope.go:122] "RemoveContainer" containerID="b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5" Oct 31 03:24:35.511694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.511677 4488 scope.go:122] "RemoveContainer" containerID="b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5" Oct 31 03:24:35.512010 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:24:35.511971 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5\": container with ID starting with b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5 not found: ID does not exist" containerID="b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5" Oct 31 03:24:35.512123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.512020 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5"} err="failed to get container status \"b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5\": rpc error: code = NotFound desc = could not find container \"b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5\": container with ID starting with b1e9585c2ad339a2c4eaee8853b07332aeab2eda5393df9b5cb2f01d71a5b0a5 not found: ID does not exist" Oct 31 03:24:35.676890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.676855 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gv22m"] Oct 31 03:24:35.681877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.681851 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gv22m"] Oct 31 03:24:35.759843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.759738 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:35.916772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:35.916732 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.146394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:36.146355 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.298421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:36.298392 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.315135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:36.315081 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad1edce-bfe0-4d59-a377-bb9818d7ec71" path="/var/lib/kubelet/pods/0ad1edce-bfe0-4d59-a377-bb9818d7ec71/volumes" Oct 31 03:24:36.450624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:36.450515 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.588613 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:24:36.588576 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:24:38.442930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.442890 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv"] Oct 31 03:24:38.443456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.443350 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0ad1edce-bfe0-4d59-a377-bb9818d7ec71" containerName="authorino" Oct 31 03:24:38.443456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.443369 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad1edce-bfe0-4d59-a377-bb9818d7ec71" containerName="authorino" Oct 31 03:24:38.443568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.443460 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad1edce-bfe0-4d59-a377-bb9818d7ec71" containerName="authorino" Oct 31 03:24:38.447748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.447724 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.500627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.500596 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 03:24:38.511140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.511111 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4db1e9-442c-4a92-8849-c25abc3b7d5e-metrics-certs\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.511318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.511229 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mxd\" (UniqueName: \"kubernetes.io/projected/da4db1e9-442c-4a92-8849-c25abc3b7d5e-kube-api-access-68mxd\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.511318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.511259 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da4db1e9-442c-4a92-8849-c25abc3b7d5e-manager-config\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.511404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.511331 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da4db1e9-442c-4a92-8849-c25abc3b7d5e-cert\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.550312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.550275 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 03:24:38.550312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.550279 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 03:24:38.550516 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.550347 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-s2zcq\"" Oct 31 03:24:38.551413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.551394 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv"] Oct 31 03:24:38.612600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.612566 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4db1e9-442c-4a92-8849-c25abc3b7d5e-metrics-certs\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.612772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.612622 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mxd\" (UniqueName: \"kubernetes.io/projected/da4db1e9-442c-4a92-8849-c25abc3b7d5e-kube-api-access-68mxd\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.612772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.612653 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da4db1e9-442c-4a92-8849-c25abc3b7d5e-manager-config\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.612772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.612680 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da4db1e9-442c-4a92-8849-c25abc3b7d5e-cert\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.613449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.613426 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da4db1e9-442c-4a92-8849-c25abc3b7d5e-manager-config\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.615260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.615244 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4db1e9-442c-4a92-8849-c25abc3b7d5e-metrics-certs\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.615326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.615311 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da4db1e9-442c-4a92-8849-c25abc3b7d5e-cert\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.674340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.674304 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mxd\" (UniqueName: \"kubernetes.io/projected/da4db1e9-442c-4a92-8849-c25abc3b7d5e-kube-api-access-68mxd\") pod \"jobset-controller-manager-755dd9975b-t5rhv\" (UID: \"da4db1e9-442c-4a92-8849-c25abc3b7d5e\") " pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:38.760372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:38.760293 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:39.137303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:39.137267 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv"] Oct 31 03:24:39.140524 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:24:39.140497 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4db1e9_442c_4a92_8849_c25abc3b7d5e.slice/crio-f65003bc3268a7107fd053aff17ad5af5598233768ae87ec57482548978f07ee WatchSource:0}: Error finding container f65003bc3268a7107fd053aff17ad5af5598233768ae87ec57482548978f07ee: Status 404 returned error can't find the container with id f65003bc3268a7107fd053aff17ad5af5598233768ae87ec57482548978f07ee Oct 31 03:24:39.142531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:39.142511 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:24:39.522298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:39.522206 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" event={"ID":"da4db1e9-442c-4a92-8849-c25abc3b7d5e","Type":"ContainerStarted","Data":"f65003bc3268a7107fd053aff17ad5af5598233768ae87ec57482548978f07ee"} Oct 31 03:24:42.238364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:42.238329 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:42.396179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:42.396144 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:44.498972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:44.498878 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gc8k4" Oct 31 03:24:44.862162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:44.862132 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:45.098292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:45.098249 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:45.555858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:45.555765 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" event={"ID":"da4db1e9-442c-4a92-8849-c25abc3b7d5e","Type":"ContainerStarted","Data":"1c429e6e44aa2bb28f53966ba518f83a71a53e6a17188306aa92cf8ad18ed8f2"} Oct 31 03:24:45.556259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:45.555892 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:24:45.673238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:45.673162 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" podStartSLOduration=1.5888084949999999 podStartE2EDuration="7.673148677s" totalImagesPullingTime="6.084340182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:45.672381557 +0000 UTC m=+313.469099819" watchObservedRunningTime="2026-10-31 03:24:45.673148677 +0000 UTC m=+313.469867011" Oct 31 03:24:48.244049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:48.244017 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:48.397343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:48.397310 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:49.136711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:49.136674 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:49.393708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:49.393638 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:50.007029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:50.006998 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:54.731773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:54.731736 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:54.888950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:54.888917 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:56.566221 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:24:56.566191 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-755dd9975b-t5rhv" Oct 31 03:25:01.290442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:01.290409 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:01.447452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:01.447419 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:03.889367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:03.889334 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:04.046199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:04.046165 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:06.461971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:06.461932 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:06.588596 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:25:06.588553 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:25:06.667011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:06.666981 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.503248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:07.503211 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.660536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:07.660502 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.894982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:07.894954 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:08.056624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:08.056592 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:08.219173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:08.219059 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.523330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:10.523290 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.692001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:10.691918 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.847826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:10.847793 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:11.016697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:11.016662 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:14.152185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:14.152135 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:14.306870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:14.306837 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:16.855247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:16.855214 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:17.024280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:17.024245 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:18.080565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:18.080529 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:18.340039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:18.339948 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.994671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:19.994628 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:20.152203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:20.152174 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:20.697472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:20.697438 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:20.849720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:20.849689 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:36.587724 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:25:36.587690 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:25:50.244539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:50.244506 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:50.573189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:50.573111 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:52.413367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:52.413332 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:52.581191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:52.581156 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:52.772956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:52.772874 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.019503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:53.019470 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:58.618039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:58.617997 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:58.775383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:58.775345 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:58.928967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:58.928876 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.088427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:59.088397 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.231392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:59.231315 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:25:59.265439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:59.265409 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.418510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:59.418469 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.635020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:59.634985 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.794461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:25:59.794435 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:00.925585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:00.925550 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:03.261056 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:03.261017 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:03.439758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:03.439729 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:04.501353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.501321 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9f94549b4-k58jq"] Oct 31 03:26:04.504923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.504906 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:04.557590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.557557 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bq6cs\"" Oct 31 03:26:04.608612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.608579 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9f94549b4-k58jq"] Oct 31 03:26:04.657683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.657658 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 03:26:04.669419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.669389 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4cs\" (UniqueName: \"kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs\") pod \"authorino-9f94549b4-k58jq\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:04.669564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.669444 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert\") pod \"authorino-9f94549b4-k58jq\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:04.770529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.770444 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert\") pod \"authorino-9f94549b4-k58jq\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:04.770529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.770513 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4cs\" (UniqueName: \"kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs\") pod \"authorino-9f94549b4-k58jq\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:04.773099 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.773073 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert\") pod \"authorino-9f94549b4-k58jq\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:04.831946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:04.831909 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4cs\" (UniqueName: \"kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs\") pod \"authorino-9f94549b4-k58jq\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:05.118612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:05.118564 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:26:05.489588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:05.489507 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9f94549b4-k58jq"] Oct 31 03:26:05.492601 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:26:05.492563 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc616b7ef_6d42_46b5_8467_edcaf3af7288.slice/crio-419b094eae3de4165be77a9171ea795bab796900ca2e37dd35a28ea495992a07 WatchSource:0}: Error finding container 419b094eae3de4165be77a9171ea795bab796900ca2e37dd35a28ea495992a07: Status 404 returned error can't find the container with id 419b094eae3de4165be77a9171ea795bab796900ca2e37dd35a28ea495992a07 Oct 31 03:26:05.884441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:05.884407 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9f94549b4-k58jq" event={"ID":"c616b7ef-6d42-46b5-8467-edcaf3af7288","Type":"ContainerStarted","Data":"419b094eae3de4165be77a9171ea795bab796900ca2e37dd35a28ea495992a07"} Oct 31 03:26:06.088483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.088448 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.246466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.246387 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.402253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.402222 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.555078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.554999 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.588968 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:26:06.588930 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:26:06.710227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.710186 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.861769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.861735 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.892288 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:06.892250 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9f94549b4-k58jq" event={"ID":"c616b7ef-6d42-46b5-8467-edcaf3af7288","Type":"ContainerStarted","Data":"486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04"} Oct 31 03:26:07.007240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:07.007187 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-9f94549b4-k58jq" podStartSLOduration=1.8002851770000001 podStartE2EDuration="3.007173324s" totalImagesPullingTime="1.206888147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:07.006527162 +0000 UTC m=+394.803245423" watchObservedRunningTime="2026-10-31 03:26:07.007173324 +0000 UTC m=+394.803891576" Oct 31 03:26:07.019620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:07.019581 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:07.180357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:07.180275 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:07.372522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:07.372490 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:18.430149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:18.430064 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:25.586773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:25.586736 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:25.743238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:26:25.743206 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:36.588103 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:26:36.588049 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:27:06.588196 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:27:06.588161 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:27:18.831810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:27:18.831778 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:36.587443 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:27:36.587400 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:27:41.325274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:27:41.325238 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:41.495061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:27:41.495026 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:42.714455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:27:42.714423 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:42.872472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:27:42.872423 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:06.053276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:06.053244 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:06.589473 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:28:06.589430 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:28:09.473698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:09.473660 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:28:13.322414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:13.322384 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:16.496115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:16.496064 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:16.910913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:16.910874 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:28:19.119054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:19.119018 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:19.274633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:19.274603 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:19.839581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:19.839546 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:23.532274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:23.532225 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:26.860109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:26.860055 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:30.072349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:30.072299 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:28:31.399816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.399772 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d"] Oct 31 03:28:31.403122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.403079 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.455532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.455506 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 03:28:31.505418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.505383 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 03:28:31.505634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.505475 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 03:28:31.505634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.505563 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8tx4\"" Oct 31 03:28:31.506715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.506695 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d"] Oct 31 03:28:31.508723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.508697 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:31.523465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.523435 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.523465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.523465 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.523653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.523561 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.523653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.523591 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.523653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.523627 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e252476b-aafe-4d02-9673-c55bbb915a52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.523794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.523764 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgnw9\" (UniqueName: \"kubernetes.io/projected/e252476b-aafe-4d02-9673-c55bbb915a52-kube-api-access-rgnw9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.624781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.624743 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.624781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.624785 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.624820 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e252476b-aafe-4d02-9673-c55bbb915a52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.624849 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgnw9\" (UniqueName: \"kubernetes.io/projected/e252476b-aafe-4d02-9673-c55bbb915a52-kube-api-access-rgnw9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.624877 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.624897 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.625245 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.625288 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.625377 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.625345 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.627267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.627247 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e252476b-aafe-4d02-9673-c55bbb915a52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.627518 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.627502 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e252476b-aafe-4d02-9673-c55bbb915a52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.686760 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.686686 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgnw9\" (UniqueName: \"kubernetes.io/projected/e252476b-aafe-4d02-9673-c55bbb915a52-kube-api-access-rgnw9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d\" (UID: \"e252476b-aafe-4d02-9673-c55bbb915a52\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:31.714755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:31.714717 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:32.091670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:32.091643 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d"] Oct 31 03:28:32.094438 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:28:32.094409 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode252476b_aafe_4d02_9673_c55bbb915a52.slice/crio-471b79dd33a174408770b0af49d47473d873d6ff38b8ab730eb8da555f4be864 WatchSource:0}: Error finding container 471b79dd33a174408770b0af49d47473d873d6ff38b8ab730eb8da555f4be864: Status 404 returned error can't find the container with id 471b79dd33a174408770b0af49d47473d873d6ff38b8ab730eb8da555f4be864 Oct 31 03:28:32.501456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:32.501365 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" event={"ID":"e252476b-aafe-4d02-9673-c55bbb915a52","Type":"ContainerStarted","Data":"471b79dd33a174408770b0af49d47473d873d6ff38b8ab730eb8da555f4be864"} Oct 31 03:28:34.512887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:34.512842 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" event={"ID":"e252476b-aafe-4d02-9673-c55bbb915a52","Type":"ContainerStarted","Data":"d02b675afb3f4bc836e40028dc8abe2ecef64f43a7eb78aca2786d76b6f64f99"} Oct 31 03:28:34.655356 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:34.655303 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podStartSLOduration=2.163933229 podStartE2EDuration="3.65528684s" totalImagesPullingTime="1.491353611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:34.654322072 +0000 UTC m=+542.451040342" watchObservedRunningTime="2026-10-31 03:28:34.65528684 +0000 UTC m=+542.452005097" Oct 31 03:28:34.715323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:34.715284 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:35.962317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:35.962285 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:36.589276 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:28:36.589232 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:28:41.095468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:41.095437 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:28:41.715047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:41.715007 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:41.727530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:41.727504 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:42.557398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:42.557367 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:28:44.479511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.479476 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj"] Oct 31 03:28:44.482943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.482925 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.535796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.535761 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:28:44.535796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.535761 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4td2m\"" Oct 31 03:28:44.541204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541178 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541229 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541250 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg9bw\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541369 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541416 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541396 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541418 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541434 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541557 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541541 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.541590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.541569 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.588927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.588885 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj"] Oct 31 03:28:44.642290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642256 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642298 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642324 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642339 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642381 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642402 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642451 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642489 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642513 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg9bw\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642736 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642785 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642849 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.642982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.642934 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.643383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.643362 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.645176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.645151 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.645347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.645327 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.703906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.703868 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.704082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.703884 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg9bw\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw\") pod \"maas-default-gateway-openshift-default-fd4b99879-kbfzj\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:44.795458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:44.795371 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:45.222181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:45.222084 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj"] Oct 31 03:28:45.224895 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:28:45.224864 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939b9f49_7813_4139_821b_950f2e92fc68.slice/crio-141b93b1f008335d9b4f3654ff3fa0a4256d14de81db7d22464ab77ecfdc15c2 WatchSource:0}: Error finding container 141b93b1f008335d9b4f3654ff3fa0a4256d14de81db7d22464ab77ecfdc15c2: Status 404 returned error can't find the container with id 141b93b1f008335d9b4f3654ff3fa0a4256d14de81db7d22464ab77ecfdc15c2 Oct 31 03:28:45.564747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:45.564660 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" event={"ID":"939b9f49-7813-4139-821b-950f2e92fc68","Type":"ContainerStarted","Data":"141b93b1f008335d9b4f3654ff3fa0a4256d14de81db7d22464ab77ecfdc15c2"} Oct 31 03:28:48.845305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:48.845275 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:28:48.845548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:48.845341 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:28:48.845548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:48.845369 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:28:49.582698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:49.582660 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" event={"ID":"939b9f49-7813-4139-821b-950f2e92fc68","Type":"ContainerStarted","Data":"2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb"} Oct 31 03:28:49.796425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:49.796391 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:49.797920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:49.797889 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:49.798052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:49.797953 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:49.803645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:49.803595 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podStartSLOduration=2.186207139 podStartE2EDuration="5.803580608s" totalImagesPullingTime="3.617373469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:49.7527111 +0000 UTC m=+557.549429360" watchObservedRunningTime="2026-10-31 03:28:49.803580608 +0000 UTC m=+557.600298866" Oct 31 03:28:50.795935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:50.795893 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:50.796407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:50.795980 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:51.796250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:51.796210 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:51.796739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:51.796293 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:52.795798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:52.795767 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:52.796002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:52.795819 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:53.795806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:53.795769 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:53.798232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:53.795828 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:53.980966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:53.980914 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:54.796462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:54.796427 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:54.796975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:54.796490 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:55.796165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:55.796129 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kbfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 31 03:28:55.796411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:55.796197 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 31 03:28:57.090757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:57.090722 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:57.091236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:57.091110 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:57.091975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:57.091924 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:28:59.602214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.602182 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6"] Oct 31 03:28:59.605502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.605484 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704395 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704449 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704617 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704655 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704733 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704764 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704780 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbmf\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704801 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.704990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.704859 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.711519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.711487 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6"] Oct 31 03:28:59.805568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805529 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805566 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805590 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805616 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805716 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbmf\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805761 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805786 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.805832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805825 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.806157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.805867 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.806157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.806000 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.806157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.806033 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.806314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.806216 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.806496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.806404 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.806637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.806617 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.808079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.808056 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.808366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.808339 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.867590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.867509 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.867590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.867546 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbmf\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf\") pod \"maas-default-gateway-openshift-default-fd4b99879-bt2j6\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:28:59.917433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:28:59.917404 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:29:00.343567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.343535 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6"] Oct 31 03:29:00.346560 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:29:00.346530 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cf0fca_bad8_48f4_95fc_34c3cee38534.slice/crio-c85c370eae93806c9662b9177083266c2d8dc8cad71d8f15f1f9486f62b33a67 WatchSource:0}: Error finding container c85c370eae93806c9662b9177083266c2d8dc8cad71d8f15f1f9486f62b33a67: Status 404 returned error can't find the container with id c85c370eae93806c9662b9177083266c2d8dc8cad71d8f15f1f9486f62b33a67 Oct 31 03:29:00.349635 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.349602 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:29:00.349708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.349669 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:29:00.349708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.349699 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:29:00.631926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.631822 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" event={"ID":"f5cf0fca-bad8-48f4-95fc-34c3cee38534","Type":"ContainerStarted","Data":"e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8"} Oct 31 03:29:00.631926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.631881 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" event={"ID":"f5cf0fca-bad8-48f4-95fc-34c3cee38534","Type":"ContainerStarted","Data":"c85c370eae93806c9662b9177083266c2d8dc8cad71d8f15f1f9486f62b33a67"} Oct 31 03:29:00.754741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.754686 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podStartSLOduration=1.754668515 podStartE2EDuration="1.754668515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:00.753572139 +0000 UTC m=+568.550290398" watchObservedRunningTime="2026-10-31 03:29:00.754668515 +0000 UTC m=+568.551386774" Oct 31 03:29:00.918453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.918316 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:29:00.921575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.921532 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6746c6f78c-5kp45"] Oct 31 03:29:00.927059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.927028 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:00.981776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:00.981739 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6746c6f78c-5kp45"] Oct 31 03:29:01.022116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.018866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d833b50c-4425-47f6-b1cc-f1dfb5154578-tls-cert\") pod \"authorino-6746c6f78c-5kp45\" (UID: \"d833b50c-4425-47f6-b1cc-f1dfb5154578\") " pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.022116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.018923 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwc8\" (UniqueName: \"kubernetes.io/projected/d833b50c-4425-47f6-b1cc-f1dfb5154578-kube-api-access-czwc8\") pod \"authorino-6746c6f78c-5kp45\" (UID: \"d833b50c-4425-47f6-b1cc-f1dfb5154578\") " pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.120771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.120716 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d833b50c-4425-47f6-b1cc-f1dfb5154578-tls-cert\") pod \"authorino-6746c6f78c-5kp45\" (UID: \"d833b50c-4425-47f6-b1cc-f1dfb5154578\") " pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.121778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.121753 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwc8\" (UniqueName: \"kubernetes.io/projected/d833b50c-4425-47f6-b1cc-f1dfb5154578-kube-api-access-czwc8\") pod \"authorino-6746c6f78c-5kp45\" (UID: \"d833b50c-4425-47f6-b1cc-f1dfb5154578\") " pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.134577 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.134553 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d833b50c-4425-47f6-b1cc-f1dfb5154578-tls-cert\") pod \"authorino-6746c6f78c-5kp45\" (UID: \"d833b50c-4425-47f6-b1cc-f1dfb5154578\") " pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.183554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.183480 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwc8\" (UniqueName: \"kubernetes.io/projected/d833b50c-4425-47f6-b1cc-f1dfb5154578-kube-api-access-czwc8\") pod \"authorino-6746c6f78c-5kp45\" (UID: \"d833b50c-4425-47f6-b1cc-f1dfb5154578\") " pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.242444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.242415 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6746c6f78c-5kp45" Oct 31 03:29:01.335136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.333953 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:01.493856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.493778 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:01.617650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.617569 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6746c6f78c-5kp45"] Oct 31 03:29:01.633321 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:29:01.633279 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd833b50c_4425_47f6_b1cc_f1dfb5154578.slice/crio-9644c526d01276b740be0a80a94903b6ac8a908e9ffae7d399e2c7edff32a270 WatchSource:0}: Error finding container 9644c526d01276b740be0a80a94903b6ac8a908e9ffae7d399e2c7edff32a270: Status 404 returned error can't find the container with id 9644c526d01276b740be0a80a94903b6ac8a908e9ffae7d399e2c7edff32a270 Oct 31 03:29:01.929296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.929255 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bt2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 31 03:29:01.929618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:01.929587 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 31 03:29:02.643188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:02.643139 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6746c6f78c-5kp45" event={"ID":"d833b50c-4425-47f6-b1cc-f1dfb5154578","Type":"ContainerStarted","Data":"9644c526d01276b740be0a80a94903b6ac8a908e9ffae7d399e2c7edff32a270"} Oct 31 03:29:02.928178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:02.926604 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bt2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 31 03:29:02.928178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:02.926680 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 31 03:29:03.649250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:03.649214 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6746c6f78c-5kp45" event={"ID":"d833b50c-4425-47f6-b1cc-f1dfb5154578","Type":"ContainerStarted","Data":"e597f91efd1f039da3e8b5242d560f5a57706d7787c3d57ac677966930ed7967"} Oct 31 03:29:03.764662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:03.764602 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6746c6f78c-5kp45" podStartSLOduration=2.21685203 podStartE2EDuration="3.764588221s" totalImagesPullingTime="1.547736191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:03.763622102 +0000 UTC m=+571.560340360" watchObservedRunningTime="2026-10-31 03:29:03.764588221 +0000 UTC m=+571.561306480" Oct 31 03:29:03.833739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:03.833685 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9f94549b4-k58jq"] Oct 31 03:29:03.833940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:03.833911 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-9f94549b4-k58jq" podUID="c616b7ef-6d42-46b5-8467-edcaf3af7288" containerName="authorino" containerID="cri-o://486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04" gracePeriod=30 Oct 31 03:29:03.920048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:03.919492 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bt2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 31 03:29:03.920048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:03.919553 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 31 03:29:04.149926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.149901 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:29:04.251465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.251381 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs\" (UniqueName: \"kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs\") pod \"c616b7ef-6d42-46b5-8467-edcaf3af7288\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " Oct 31 03:29:04.251465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.251451 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert\" (UniqueName: \"kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert\") pod \"c616b7ef-6d42-46b5-8467-edcaf3af7288\" (UID: \"c616b7ef-6d42-46b5-8467-edcaf3af7288\") " Oct 31 03:29:04.253653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.253622 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs" pod "c616b7ef-6d42-46b5-8467-edcaf3af7288" (UID: "c616b7ef-6d42-46b5-8467-edcaf3af7288"). InnerVolumeSpecName "kube-api-access-dc4cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:29:04.263311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.263279 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert" pod "c616b7ef-6d42-46b5-8467-edcaf3af7288" (UID: "c616b7ef-6d42-46b5-8467-edcaf3af7288"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:29:04.352505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.352281 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4cs\" (UniqueName: \"kubernetes.io/projected/c616b7ef-6d42-46b5-8467-edcaf3af7288-kube-api-access-dc4cs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:29:04.352505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.352312 4488 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c616b7ef-6d42-46b5-8467-edcaf3af7288-tls-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:29:04.655647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.655605 4488 generic.go:356] "Generic (PLEG): container finished" podID="c616b7ef-6d42-46b5-8467-edcaf3af7288" containerID="486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04" exitCode=0 Oct 31 03:29:04.656147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.655655 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9f94549b4-k58jq" Oct 31 03:29:04.656147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.655647 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9f94549b4-k58jq" event={"ID":"c616b7ef-6d42-46b5-8467-edcaf3af7288","Type":"ContainerDied","Data":"486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04"} Oct 31 03:29:04.656147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.655764 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9f94549b4-k58jq" event={"ID":"c616b7ef-6d42-46b5-8467-edcaf3af7288","Type":"ContainerDied","Data":"419b094eae3de4165be77a9171ea795bab796900ca2e37dd35a28ea495992a07"} Oct 31 03:29:04.656147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.655781 4488 scope.go:122] "RemoveContainer" containerID="486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04" Oct 31 03:29:04.664713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.664688 4488 scope.go:122] "RemoveContainer" containerID="486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04" Oct 31 03:29:04.665027 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:29:04.665007 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04\": container with ID starting with 486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04 not found: ID does not exist" containerID="486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04" Oct 31 03:29:04.665077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.665038 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04"} err="failed to get container status \"486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04\": rpc error: code = NotFound desc = could not find container \"486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04\": container with ID starting with 486ae9d6c029163396d0ed2fe10c0c7df49e89cbbe5bae0974d1483e89941b04 not found: ID does not exist" Oct 31 03:29:04.836274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.836241 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9f94549b4-k58jq"] Oct 31 03:29:04.840510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.840484 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9f94549b4-k58jq"] Oct 31 03:29:04.924705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.924609 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bt2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 31 03:29:04.924974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:04.924935 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 31 03:29:05.921562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:05.921525 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bt2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 31 03:29:05.922167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:05.922132 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 31 03:29:06.315654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:06.315563 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c616b7ef-6d42-46b5-8467-edcaf3af7288" path="/var/lib/kubelet/pods/c616b7ef-6d42-46b5-8467-edcaf3af7288/volumes" Oct 31 03:29:06.592244 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:29:06.592203 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:29:06.921178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:06.920068 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bt2j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 31 03:29:06.921178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:06.920184 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 31 03:29:08.623059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:08.623019 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:29:08.623570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:08.623536 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:29:08.627264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:08.627241 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:29:09.397039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:09.397006 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:09.560225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:09.560194 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:10.929640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:10.929603 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:11.084357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.084314 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:11.314968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.314885 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2"] Oct 31 03:29:11.315256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.315244 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c616b7ef-6d42-46b5-8467-edcaf3af7288" containerName="authorino" Oct 31 03:29:11.315308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.315258 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c616b7ef-6d42-46b5-8467-edcaf3af7288" containerName="authorino" Oct 31 03:29:11.315342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.315324 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c616b7ef-6d42-46b5-8467-edcaf3af7288" containerName="authorino" Oct 31 03:29:11.319459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.319438 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.371967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.371920 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 03:29:11.372144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.371979 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mp6g4\"" Oct 31 03:29:11.420923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.420880 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ba0a01-b535-458e-b03b-63a5f48eeaeb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zqpt2\" (UID: \"f5ba0a01-b535-458e-b03b-63a5f48eeaeb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.421230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.421194 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45mw\" (UniqueName: \"kubernetes.io/projected/f5ba0a01-b535-458e-b03b-63a5f48eeaeb-kube-api-access-m45mw\") pod \"payload-pre-processing-7b98bb678c-zqpt2\" (UID: \"f5ba0a01-b535-458e-b03b-63a5f48eeaeb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.423189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.423170 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2"] Oct 31 03:29:11.522156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.522120 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ba0a01-b535-458e-b03b-63a5f48eeaeb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zqpt2\" (UID: \"f5ba0a01-b535-458e-b03b-63a5f48eeaeb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.522156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.522161 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45mw\" (UniqueName: \"kubernetes.io/projected/f5ba0a01-b535-458e-b03b-63a5f48eeaeb-kube-api-access-m45mw\") pod \"payload-pre-processing-7b98bb678c-zqpt2\" (UID: \"f5ba0a01-b535-458e-b03b-63a5f48eeaeb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.523168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.523132 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ba0a01-b535-458e-b03b-63a5f48eeaeb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zqpt2\" (UID: \"f5ba0a01-b535-458e-b03b-63a5f48eeaeb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.583753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.583717 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45mw\" (UniqueName: \"kubernetes.io/projected/f5ba0a01-b535-458e-b03b-63a5f48eeaeb-kube-api-access-m45mw\") pod \"payload-pre-processing-7b98bb678c-zqpt2\" (UID: \"f5ba0a01-b535-458e-b03b-63a5f48eeaeb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:11.631583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:11.631534 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:12.019811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:12.019728 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2"] Oct 31 03:29:12.023248 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:29:12.023213 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ba0a01_b535_458e_b03b_63a5f48eeaeb.slice/crio-f5c927e143421a99f3bc71df910cb8e98bfa96e20c305c3e5da065fbb930412c WatchSource:0}: Error finding container f5c927e143421a99f3bc71df910cb8e98bfa96e20c305c3e5da065fbb930412c: Status 404 returned error can't find the container with id f5c927e143421a99f3bc71df910cb8e98bfa96e20c305c3e5da065fbb930412c Oct 31 03:29:12.697253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:12.697214 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" event={"ID":"f5ba0a01-b535-458e-b03b-63a5f48eeaeb","Type":"ContainerStarted","Data":"f5c927e143421a99f3bc71df910cb8e98bfa96e20c305c3e5da065fbb930412c"} Oct 31 03:29:15.712286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:15.712247 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" event={"ID":"f5ba0a01-b535-458e-b03b-63a5f48eeaeb","Type":"ContainerStarted","Data":"2e22a7d2545a88eedbc0069baf9623bce5c91646573e0a54ecb8838a49514fec"} Oct 31 03:29:15.712874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:15.712390 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:15.830237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:15.830185 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" podStartSLOduration=1.976600848 podStartE2EDuration="4.830166808s" totalImagesPullingTime="2.85356596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:15.828738215 +0000 UTC m=+583.625456477" watchObservedRunningTime="2026-10-31 03:29:15.830166808 +0000 UTC m=+583.626885066" Oct 31 03:29:19.464322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:19.464279 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:26.720289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:26.720195 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zqpt2" Oct 31 03:29:33.768730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:33.768697 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:34.312304 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:34.312265 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:36.033212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:36.033174 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:36.216334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:36.216258 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:36.587077 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:29:36.587036 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:29:37.787003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:37.786966 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:37.970220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:29:37.970187 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:06.587263 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:30:06.587225 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:30:11.589243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:11.589208 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:12.641835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:12.641767 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:19.563165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:19.563069 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:19.727497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:19.727465 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:19.891986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:19.891954 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:20.094457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.094425 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl"] Oct 31 03:30:20.097829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.097809 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142399 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142443 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142566 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142627 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142686 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142706 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142731 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142750 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsfv\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.142998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.142785 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.150459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.150433 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b63425-openshift-default-dockercfg-frs87\"" Oct 31 03:30:20.195662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.195630 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:20.203188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.203156 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl"] Oct 31 03:30:20.243444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243406 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243450 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243469 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243483 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243502 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsfv\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243551 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243630 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243657 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.243691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243688 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.244043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243878 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.244043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.243910 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.244170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.244064 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.244482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.244443 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.244737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.244518 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.246366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.246348 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.246448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.246364 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.309822 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.309789 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.309971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.309892 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsfv\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.378788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.378753 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:20.410776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.410192 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:20.411786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.411761 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98"] Oct 31 03:30:20.414909 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.414889 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.446581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446545 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqwq\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.446581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446583 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.446817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446600 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.446817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446665 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.446817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446734 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.446817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446770 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.447028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446828 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.447028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446873 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.447028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.446908 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.469456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.469195 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e00ad3-openshift-default-dockercfg-s9kbq\"" Oct 31 03:30:20.478229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.478131 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98"] Oct 31 03:30:20.539058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.539017 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:20.547790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547749 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547798 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547829 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547862 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqwq\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547890 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547920 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547945 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.547996 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.548037 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.548362 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.548459 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.548666 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.548893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.548836 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.549042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.549020 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.550455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.550435 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.550792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.550770 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.610284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.610250 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.610687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.610414 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqwq\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.738933 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.738839 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:20.829550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.829505 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:20.843371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.843334 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46"] Oct 31 03:30:20.847426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.847402 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.870188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.870071 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl"] Oct 31 03:30:20.872417 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:20.872391 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb948593_f56f_46ec_b0a7_6ffa5c8c9dc1.slice/crio-9191f1a4312fbc031a870d21fa0e82a7156f08844cf4dcb1c13def419e513f27 WatchSource:0}: Error finding container 9191f1a4312fbc031a870d21fa0e82a7156f08844cf4dcb1c13def419e513f27: Status 404 returned error can't find the container with id 9191f1a4312fbc031a870d21fa0e82a7156f08844cf4dcb1c13def419e513f27 Oct 31 03:30:20.874490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.874460 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:20.874553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.874524 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:20.874596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.874553 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:20.907520 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.907478 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-096812-openshift-default-dockercfg-7vgck\"" Oct 31 03:30:20.910220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.910070 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46"] Oct 31 03:30:20.952367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952328 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952492 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952529 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952563 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952634 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952747 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952847 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952933 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952905 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.952986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.952940 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:20.988601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.988564 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" event={"ID":"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1","Type":"ContainerStarted","Data":"aa7cf2c650ab5a4a840611bb3f2b5497518393d4e4a3e9b95b2f42415710d1d7"} Oct 31 03:30:20.988764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:20.988633 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" event={"ID":"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1","Type":"ContainerStarted","Data":"9191f1a4312fbc031a870d21fa0e82a7156f08844cf4dcb1c13def419e513f27"} Oct 31 03:30:21.000541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.000426 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.054273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054228 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054284 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054452 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054593 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054565 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054592 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054627 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054653 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054703 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054761 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.054907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.054844 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.055127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.055070 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.055218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.055133 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.055267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.055236 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.055734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.055714 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.057341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.057323 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.057341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.057337 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.114681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.114622 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podStartSLOduration=1.114607167 podStartE2EDuration="1.114607167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.112235797 +0000 UTC m=+648.908954055" watchObservedRunningTime="2026-10-31 03:30:21.114607167 +0000 UTC m=+648.911325424" Oct 31 03:30:21.120704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.120682 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.120909 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.120883 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-kfp46\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.123767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.123709 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67"] Oct 31 03:30:21.128775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.128753 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98"] Oct 31 03:30:21.128887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.128850 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.130521 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:21.130497 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf2fbea_df19_478b_b3b4_d4e86fa891c6.slice/crio-f714de5d18f6d4c30539da1b119dc8e68beefa5e14252abf649b5bca9ae936af WatchSource:0}: Error finding container f714de5d18f6d4c30539da1b119dc8e68beefa5e14252abf649b5bca9ae936af: Status 404 returned error can't find the container with id f714de5d18f6d4c30539da1b119dc8e68beefa5e14252abf649b5bca9ae936af Oct 31 03:30:21.132633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.132593 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.132715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.132684 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.132756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.132726 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.155607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155574 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.155746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155651 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.155746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155708 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.155813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155767 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.155813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155808 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.155876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155852 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.155993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.155975 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9rw\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.156035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.156024 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.156143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.156123 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.161552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.161513 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:21.181662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.181624 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b98893-openshift-default-dockercfg-f7nhz\"" Oct 31 03:30:21.183410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.183370 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.191263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.191229 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67"] Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257323 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257383 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9rw\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257425 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257457 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257509 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257554 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257594 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257622 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.258182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.257657 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.260448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.259688 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.260448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.259714 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.260448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.260062 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.260448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.260397 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.261133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.261078 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.261765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.261724 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.264165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.264120 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.320756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.320722 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.320756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.320751 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9rw\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.343317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.343285 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.411139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.411080 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:21.416546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.416520 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:21.442829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.442798 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:21.447684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.447582 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn"] Oct 31 03:30:21.451596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.451572 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.508494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.505780 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-dec4a0-openshift-default-dockercfg-5rrgk\"" Oct 31 03:30:21.511543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.510664 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.515651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.513485 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn"] Oct 31 03:30:21.561201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561171 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561212 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561360 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561397 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561461 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561485 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4579f\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561514 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561549 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.561660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.561578 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.562126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.562108 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46"] Oct 31 03:30:21.562935 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:21.562777 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c44308_7237_4f0a_8011_6ca53cf7c381.slice/crio-a881399a815c816e830cddacd57faf5f0520042cd6124c1e75ec69ef269fc812 WatchSource:0}: Error finding container a881399a815c816e830cddacd57faf5f0520042cd6124c1e75ec69ef269fc812: Status 404 returned error can't find the container with id a881399a815c816e830cddacd57faf5f0520042cd6124c1e75ec69ef269fc812 Oct 31 03:30:21.565790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.565751 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.565892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.565835 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.565892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.565872 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.662862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.662814 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.662862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.662861 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.662917 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.662942 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4579f\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.662969 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663004 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663033 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663126 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663154 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663238 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663468 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663518 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.663837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.663667 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.664438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.664417 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.665809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.665784 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.666190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.666167 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.671898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.671875 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.725214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.725174 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.725363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.725278 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4579f\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.773718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.773631 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh"] Oct 31 03:30:21.778352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.778324 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:21.778536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.778520 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.829672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.829581 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.831517 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.830834 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-14cb5a-openshift-default-dockercfg-hg7j6\"" Oct 31 03:30:21.843689 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.843655 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh"] Oct 31 03:30:21.848146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.848080 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67"] Oct 31 03:30:21.851807 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:21.851774 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1eaf11_4e83_46a3_aec8_2f692d367be3.slice/crio-f5eed4d0171df58feca7b59da31a95d0df947a64f7163b485dc4cd29edc53707 WatchSource:0}: Error finding container f5eed4d0171df58feca7b59da31a95d0df947a64f7163b485dc4cd29edc53707: Status 404 returned error can't find the container with id f5eed4d0171df58feca7b59da31a95d0df947a64f7163b485dc4cd29edc53707 Oct 31 03:30:21.854626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.854577 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.854738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.854677 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.854738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.854722 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:21.864532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.864494 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.864703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.864683 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.864816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.864799 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.864916 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.864883 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.864975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.864938 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.865064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.865037 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj7b\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.865157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.865103 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.865228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.865211 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.865281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.865255 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.965962 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966112 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966247 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966302 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966351 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966384 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966432 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj7b\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966465 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.966609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.966545 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.967787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.967460 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.967787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.967712 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.968033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.967968 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.968481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.968459 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.968980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.968956 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.971311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.971288 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.971414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.971372 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:21.993961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.993930 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.997133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.996971 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" event={"ID":"ab1eaf11-4e83-46a3-aec8-2f692d367be3","Type":"ContainerStarted","Data":"d5a9a81da646beb296b777f2587e39e9464ac603bbd84217081f798d6966dbd7"} Oct 31 03:30:21.997133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.997073 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" event={"ID":"ab1eaf11-4e83-46a3-aec8-2f692d367be3","Type":"ContainerStarted","Data":"f5eed4d0171df58feca7b59da31a95d0df947a64f7163b485dc4cd29edc53707"} Oct 31 03:30:21.998982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.998956 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" event={"ID":"17c44308-7237-4f0a-8011-6ca53cf7c381","Type":"ContainerStarted","Data":"ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1"} Oct 31 03:30:21.999186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:21.999169 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" event={"ID":"17c44308-7237-4f0a-8011-6ca53cf7c381","Type":"ContainerStarted","Data":"a881399a815c816e830cddacd57faf5f0520042cd6124c1e75ec69ef269fc812"} Oct 31 03:30:22.000758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.000735 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" event={"ID":"daf2fbea-df19-478b-b3b4-d4e86fa891c6","Type":"ContainerStarted","Data":"5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72"} Oct 31 03:30:22.000865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.000778 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" event={"ID":"daf2fbea-df19-478b-b3b4-d4e86fa891c6","Type":"ContainerStarted","Data":"f714de5d18f6d4c30539da1b119dc8e68beefa5e14252abf649b5bca9ae936af"} Oct 31 03:30:22.001064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.001043 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:22.002800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.002783 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:22.030317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.030242 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj7b\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:22.030935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.030894 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:22.092512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.092480 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:22.156156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.156070 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:22.162206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.162176 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:22.164396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.164346 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podStartSLOduration=1.164325315 podStartE2EDuration="1.164325315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:22.121616797 +0000 UTC m=+649.918335056" watchObservedRunningTime="2026-10-31 03:30:22.164325315 +0000 UTC m=+649.961043575" Oct 31 03:30:22.167329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.166402 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn"] Oct 31 03:30:22.171549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.171300 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:22.172366 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:22.172342 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb50f54_f39c_4cac_bb0b_039339b0c778.slice/crio-eb0aebe6a00578d868e4a4afec32dbd610748cc465c55517343f522b69a90fee WatchSource:0}: Error finding container eb0aebe6a00578d868e4a4afec32dbd610748cc465c55517343f522b69a90fee: Status 404 returned error can't find the container with id eb0aebe6a00578d868e4a4afec32dbd610748cc465c55517343f522b69a90fee Oct 31 03:30:22.175423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.174999 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:22.175423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.175062 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:22.175423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.175147 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:22.312660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.312564 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:22.381353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.381292 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podStartSLOduration=2.381272695 podStartE2EDuration="2.381272695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:22.378969114 +0000 UTC m=+650.175687373" watchObservedRunningTime="2026-10-31 03:30:22.381272695 +0000 UTC m=+650.177990955" Oct 31 03:30:22.443212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.443183 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:22.448372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.448345 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:22.466306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.466278 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:22.499969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.499775 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh"] Oct 31 03:30:22.504781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.503921 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podStartSLOduration=2.503895071 podStartE2EDuration="2.503895071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:22.501056806 +0000 UTC m=+650.297775067" watchObservedRunningTime="2026-10-31 03:30:22.503895071 +0000 UTC m=+650.300613326" Oct 31 03:30:22.507291 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:22.507257 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a18583d_f061_46bb_8af6_9b34518d4d95.slice/crio-8199c6e8e1c3f330c0920c925ab4309c0e75a4970e1ac4d0649654afa93374ec WatchSource:0}: Error finding container 8199c6e8e1c3f330c0920c925ab4309c0e75a4970e1ac4d0649654afa93374ec: Status 404 returned error can't find the container with id 8199c6e8e1c3f330c0920c925ab4309c0e75a4970e1ac4d0649654afa93374ec Oct 31 03:30:22.510645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.510598 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:22.510752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.510712 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:22.510820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.510763 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:22.740027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.739992 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:22.745435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:22.745409 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:23.007053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.007014 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" event={"ID":"6a18583d-f061-46bb-8af6-9b34518d4d95","Type":"ContainerStarted","Data":"efe25d3456b6e8dc2aa5c530ca14de386cf4b62538ad571a659f6e1166321c5e"} Oct 31 03:30:23.007263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.007079 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" event={"ID":"6a18583d-f061-46bb-8af6-9b34518d4d95","Type":"ContainerStarted","Data":"8199c6e8e1c3f330c0920c925ab4309c0e75a4970e1ac4d0649654afa93374ec"} Oct 31 03:30:23.008721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.008687 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" event={"ID":"6eb50f54-f39c-4cac-bb0b-039339b0c778","Type":"ContainerStarted","Data":"006a8ea9486d30db3754ca42dff98287bd0c835d4ef209be5582687a405a7179"} Oct 31 03:30:23.008862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.008737 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" event={"ID":"6eb50f54-f39c-4cac-bb0b-039339b0c778","Type":"ContainerStarted","Data":"eb0aebe6a00578d868e4a4afec32dbd610748cc465c55517343f522b69a90fee"} Oct 31 03:30:23.009822 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.009799 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:23.009957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.009838 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:23.009957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.009862 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:23.009957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.009898 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:30:23.009957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.009928 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:30:23.010613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.010598 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:30:23.093731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.093700 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:23.099083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.099057 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:23.128649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.128597 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podStartSLOduration=2.128582311 podStartE2EDuration="2.128582311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:23.127591633 +0000 UTC m=+650.924309891" watchObservedRunningTime="2026-10-31 03:30:23.128582311 +0000 UTC m=+650.925300570" Oct 31 03:30:23.250400 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.250299 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podStartSLOduration=2.25028016 podStartE2EDuration="2.25028016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:23.249777466 +0000 UTC m=+651.046495725" watchObservedRunningTime="2026-10-31 03:30:23.25028016 +0000 UTC m=+651.046998418" Oct 31 03:30:23.779184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.779139 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:23.784277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:23.784251 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:24.014256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.014217 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:24.014801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.014775 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:24.015285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.015267 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:30:24.015576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.015563 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:30:24.092992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.092962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:24.297549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.297516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:24.459861 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:24.459773 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:25.420211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:25.420181 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:25.591451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:25.591412 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:25.747115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:25.747002 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:27.277828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:27.277785 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:27.448774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:27.448729 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:27.607381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:27.607343 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:27.823002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:27.822969 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:27.981170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:27.981046 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:28.137650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:28.137613 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:28.291013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:28.290899 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:28.583126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:28.583081 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:28.745277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:28.745247 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:28.897527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:28.897429 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:29.222913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:29.222822 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:29.472528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:29.472497 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:29.653316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:29.653284 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:30.761198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:30.761164 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:30.936376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:30.936346 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:31.109449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:31.109409 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:31.713581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:31.713541 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:31.885903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:31.885872 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:32.055644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:32.055545 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:35.004211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:35.004156 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:35.187739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:35.187703 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:36.150855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:36.150816 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:36.589153 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:30:36.589108 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:30:37.659045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:37.659008 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:37.838480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:37.838445 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:38.996073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:38.996035 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:30:41.453190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:41.453152 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:41.622883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:41.622825 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:41.776118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:41.775964 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:42.496924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:42.496884 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:42.652676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:42.652639 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:46.798592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:46.798560 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:46.959431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:46.959398 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:47.119652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.119611 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:47.349468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.349434 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:47.523374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.523241 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:47.893124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.893070 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd"] Oct 31 03:30:47.897278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.897252 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:47.929687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.929651 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:30:47.973122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.973073 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-14cb5a-dockercfg-snl9r\"" Oct 31 03:30:47.975900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.975251 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-14cb5a\"" Oct 31 03:30:47.976229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:47.975790 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd"] Oct 31 03:30:48.084501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.084464 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxgq\" (UniqueName: \"kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq\") pod \"payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.084702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.084526 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.186069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.185974 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxgq\" (UniqueName: \"kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq\") pod \"payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.186243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.186071 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.186860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.186840 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.248049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.247998 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxgq\" (UniqueName: \"kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq\") pod \"payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.521706 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.521677 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:48.895931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:48.895891 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd"] Oct 31 03:30:48.898794 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:48.898766 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c61412_24c1_4932_8e68_b3f4720c5f77.slice/crio-42fc96e297029d3e95ca60410b9f83d989bf72466ec4b3a98172b51dbdd6e3a2 WatchSource:0}: Error finding container 42fc96e297029d3e95ca60410b9f83d989bf72466ec4b3a98172b51dbdd6e3a2: Status 404 returned error can't find the container with id 42fc96e297029d3e95ca60410b9f83d989bf72466ec4b3a98172b51dbdd6e3a2 Oct 31 03:30:49.156340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.156247 4488 generic.go:356] "Generic (PLEG): container finished" podID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerID="aa7cf2c650ab5a4a840611bb3f2b5497518393d4e4a3e9b95b2f42415710d1d7" exitCode=0 Oct 31 03:30:49.156340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.156321 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" event={"ID":"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1","Type":"ContainerDied","Data":"aa7cf2c650ab5a4a840611bb3f2b5497518393d4e4a3e9b95b2f42415710d1d7"} Oct 31 03:30:49.156815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.156792 4488 scope.go:122] "RemoveContainer" containerID="aa7cf2c650ab5a4a840611bb3f2b5497518393d4e4a3e9b95b2f42415710d1d7" Oct 31 03:30:49.158025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.157996 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" event={"ID":"26c61412-24c1-4932-8e68-b3f4720c5f77","Type":"ContainerStarted","Data":"9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9"} Oct 31 03:30:49.158153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.158041 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" event={"ID":"26c61412-24c1-4932-8e68-b3f4720c5f77","Type":"ContainerStarted","Data":"42fc96e297029d3e95ca60410b9f83d989bf72466ec4b3a98172b51dbdd6e3a2"} Oct 31 03:30:49.158153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.158083 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:30:49.158275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.158235 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:49.158321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.158298 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:49.158369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.158330 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:49.498430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:49.498317 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" podStartSLOduration=2.498300144 podStartE2EDuration="2.498300144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:49.497260723 +0000 UTC m=+677.293978982" watchObservedRunningTime="2026-10-31 03:30:49.498300144 +0000 UTC m=+677.295018402" Oct 31 03:30:50.164423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:50.164390 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" event={"ID":"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1","Type":"ContainerStarted","Data":"9b7ee91847064ceaad770c9d0cc2c8730bac4713b92d0c9fd6329ce4c872a8bd"} Oct 31 03:30:50.416540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:50.416433 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:50.416814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:50.416710 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:30:50.416991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:50.416957 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:30:51.413137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:51.412479 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:30:51.413137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:51.412543 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:30:52.001912 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:52.001876 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:52.030875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:52.030839 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:52.275870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:52.275781 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:52.421655 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:52.421618 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:30:52.422369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:52.422235 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:30:52.455878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:52.455831 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:53.412556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:53.412513 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:30:53.412827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:53.412796 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:30:54.423681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:54.423642 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:30:54.424400 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:54.424346 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 03:30:54.830667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:54.830616 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:54.993208 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:54.993173 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:55.193111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.193001 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:30:55.222822 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.222789 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2"] Oct 31 03:30:55.225805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.225784 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.279523 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.279486 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-909365\"" Oct 31 03:30:55.279748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.279628 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-909365-dockercfg-nzxt2\"" Oct 31 03:30:55.307120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.305869 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2"] Oct 31 03:30:55.330118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.318411 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4blw\" (UniqueName: \"kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw\") pod \"payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.330118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.318463 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.420028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.419981 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4blw\" (UniqueName: \"kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw\") pod \"payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.420354 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.420327 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.421362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.421334 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.484156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.483451 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4blw\" (UniqueName: \"kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw\") pod \"payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.530274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.530243 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:55.533179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.533139 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:30:55.539606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.539561 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:55.923826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:55.923643 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2"] Oct 31 03:30:55.926674 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:30:55.926645 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd32d418_67ed_4d6b_bacf_8b27be0664e5.slice/crio-8687ee7001105119619fadb017592fd5ac784e672ae5a5f5870632f3884a166e WatchSource:0}: Error finding container 8687ee7001105119619fadb017592fd5ac784e672ae5a5f5870632f3884a166e: Status 404 returned error can't find the container with id 8687ee7001105119619fadb017592fd5ac784e672ae5a5f5870632f3884a166e Oct 31 03:30:56.195395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:56.195301 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" event={"ID":"fd32d418-67ed-4d6b-bacf-8b27be0664e5","Type":"ContainerStarted","Data":"fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7"} Oct 31 03:30:56.195395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:56.195364 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" event={"ID":"fd32d418-67ed-4d6b-bacf-8b27be0664e5","Type":"ContainerStarted","Data":"8687ee7001105119619fadb017592fd5ac784e672ae5a5f5870632f3884a166e"} Oct 31 03:30:56.195620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:56.195578 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:30:56.753307 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:56.753277 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:56.930306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:56.930262 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:57.089333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:57.089286 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:59.235664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.235616 4488 generic.go:356] "Generic (PLEG): container finished" podID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerID="5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72" exitCode=0 Oct 31 03:30:59.236157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.235690 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" event={"ID":"daf2fbea-df19-478b-b3b4-d4e86fa891c6","Type":"ContainerDied","Data":"5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72"} Oct 31 03:30:59.236237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.236219 4488 scope.go:122] "RemoveContainer" containerID="5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72" Oct 31 03:30:59.244541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.238255 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:59.244541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.238429 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:59.244541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.238522 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:30:59.281473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.281442 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:59.446874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.446803 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" podStartSLOduration=4.446783168 podStartE2EDuration="4.446783168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:56.32455566 +0000 UTC m=+684.121273901" watchObservedRunningTime="2026-10-31 03:30:59.446783168 +0000 UTC m=+687.243501427" Oct 31 03:30:59.936397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:30:59.935390 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:00.103675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:00.103593 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:00.167706 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:00.167680 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 03:31:00.256172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:00.256009 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" event={"ID":"daf2fbea-df19-478b-b3b4-d4e86fa891c6","Type":"ContainerStarted","Data":"ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902"} Oct 31 03:31:00.740115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:00.739503 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:31:00.740115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:00.739644 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:31:00.740115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:00.739687 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 03:31:01.739693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:01.739644 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:31:01.740280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:01.739716 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 03:31:02.329362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:02.329309 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:02.528122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:02.528064 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:02.740381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:02.740300 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:31:02.740381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:02.740354 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 03:31:03.023179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:03.020807 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:03.177008 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:03.176966 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:03.740138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:03.740067 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:31:03.740358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:03.740151 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 03:31:04.282865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:04.282833 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:31:04.739535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:04.739494 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:31:04.739728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:04.739565 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 03:31:05.467063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.467024 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr"] Oct 31 03:31:05.470508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.470487 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.524431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.524386 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"openshift-service-ca.crt\"" Oct 31 03:31:05.558804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.558142 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.558804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.558318 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdk7\" (UniqueName: \"kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.558804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.558349 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.558804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.558446 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.558804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.558472 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.558804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.558506 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.574394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.574366 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"kube-root-ca.crt\"" Oct 31 03:31:05.574394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.574366 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"default-dockercfg-5rnfw\"" Oct 31 03:31:05.574644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.574468 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"simulator-w6-b63425-kserve-self-signed-certs\"" Oct 31 03:31:05.575005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.574979 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:05.576609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.576585 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr"] Oct 31 03:31:05.659494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.659457 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.659688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.659509 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdk7\" (UniqueName: \"kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.659688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.659531 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.659688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.659587 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.659688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.659616 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.659886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.659779 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.660149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.660127 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.660266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.660152 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.660266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.660239 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.662241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.662222 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.662328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.662277 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.721034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.720959 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdk7\" (UniqueName: \"kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7\") pod \"simulator-w6-b63425-kserve-5c474c9f5f-mvlqr\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:05.744414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.744384 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:31:05.744700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.744676 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:31:05.745552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.745534 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:31:05.785424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:05.785392 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:06.161203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.161162 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr"] Oct 31 03:31:06.164741 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:31:06.164710 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2958864_adde_4e3c_9ad8_48a47fb83f34.slice/crio-5e3fe5c9d32e88193acd9b26150efc53aa69ca5a29aa1e8b3f8472e914540ed7 WatchSource:0}: Error finding container 5e3fe5c9d32e88193acd9b26150efc53aa69ca5a29aa1e8b3f8472e914540ed7: Status 404 returned error can't find the container with id 5e3fe5c9d32e88193acd9b26150efc53aa69ca5a29aa1e8b3f8472e914540ed7 Oct 31 03:31:06.292756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.292720 4488 generic.go:356] "Generic (PLEG): container finished" podID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerID="d5a9a81da646beb296b777f2587e39e9464ac603bbd84217081f798d6966dbd7" exitCode=0 Oct 31 03:31:06.292950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.292801 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" event={"ID":"ab1eaf11-4e83-46a3-aec8-2f692d367be3","Type":"ContainerDied","Data":"d5a9a81da646beb296b777f2587e39e9464ac603bbd84217081f798d6966dbd7"} Oct 31 03:31:06.293320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.293302 4488 scope.go:122] "RemoveContainer" containerID="d5a9a81da646beb296b777f2587e39e9464ac603bbd84217081f798d6966dbd7" Oct 31 03:31:06.294436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.294413 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" event={"ID":"a2958864-adde-4e3c-9ad8-48a47fb83f34","Type":"ContainerStarted","Data":"fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa"} Oct 31 03:31:06.294538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.294445 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" event={"ID":"a2958864-adde-4e3c-9ad8-48a47fb83f34","Type":"ContainerStarted","Data":"5e3fe5c9d32e88193acd9b26150efc53aa69ca5a29aa1e8b3f8472e914540ed7"} Oct 31 03:31:06.294668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.294640 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:06.294732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.294707 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:06.294785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.294745 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:06.542531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.542427 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podStartSLOduration=1.542412852 podStartE2EDuration="1.542412852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.540597592 +0000 UTC m=+694.337315851" watchObservedRunningTime="2026-10-31 03:31:06.542412852 +0000 UTC m=+694.339131109" Oct 31 03:31:06.589670 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:31:06.589635 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:31:06.785667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:06.785632 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:07.204035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.204004 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:31:07.300795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.300763 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" event={"ID":"ab1eaf11-4e83-46a3-aec8-2f692d367be3","Type":"ContainerStarted","Data":"8abfdd3a6a9e73a19464b505a4c794fb6ad1f93664d7d4df1886873a26f1d271"} Oct 31 03:31:07.443311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.443264 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:31:07.444611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.444579 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:31:07.444750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.444639 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 03:31:07.830686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.830638 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:07.988665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:07.988598 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:08.019230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:08.019192 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:31:08.443300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:08.443266 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:31:08.443490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:08.443322 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 03:31:09.443277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:09.443245 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:31:09.443759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:09.443297 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 03:31:10.443535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:10.443491 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:31:10.443993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:10.443549 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 03:31:11.124115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:11.123944 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:31:11.443563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:11.443480 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:31:11.443563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:11.443539 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 03:31:12.498071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:12.498039 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:31:12.501198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:12.501169 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 03:31:13.355281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:13.355250 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:15.786343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:15.786307 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:15.799320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:15.799295 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:16.379347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:16.379312 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:31:18.364114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:18.364020 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:18.526303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:18.526262 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:18.692459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:18.692363 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:20.166736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:20.166700 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:20.326115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:20.325663 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:21.396107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:21.396058 4488 generic.go:356] "Generic (PLEG): container finished" podID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerID="ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1" exitCode=0 Oct 31 03:31:21.396628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:21.396155 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" event={"ID":"17c44308-7237-4f0a-8011-6ca53cf7c381","Type":"ContainerDied","Data":"ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1"} Oct 31 03:31:21.396628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:21.396591 4488 scope.go:122] "RemoveContainer" containerID="ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1" Oct 31 03:31:21.398036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:21.397990 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:21.398196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:21.398082 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:21.398196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:21.398156 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:22.437339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:22.437300 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" event={"ID":"17c44308-7237-4f0a-8011-6ca53cf7c381","Type":"ContainerStarted","Data":"a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb"} Oct 31 03:31:23.019120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:23.013226 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:31:23.162334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:23.162291 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:31:23.163134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:23.163109 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:31:23.163247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:23.163153 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 03:31:23.502716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:23.502677 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:24.162138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:24.162105 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:31:24.162383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:24.162360 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 03:31:25.162219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:25.162080 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:31:25.162219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:25.162166 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 03:31:26.162588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:26.162553 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:31:26.163149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:26.162633 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 03:31:27.162459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:27.162424 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:31:27.162646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:27.162485 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 03:31:27.625596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:27.625554 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:31:28.167542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:28.167507 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:31:28.168888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:28.168867 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:31:28.345964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:28.345925 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:29.492001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:29.491969 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerID="efe25d3456b6e8dc2aa5c530ca14de386cf4b62538ad571a659f6e1166321c5e" exitCode=0 Oct 31 03:31:29.492612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:29.492026 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" event={"ID":"6a18583d-f061-46bb-8af6-9b34518d4d95","Type":"ContainerDied","Data":"efe25d3456b6e8dc2aa5c530ca14de386cf4b62538ad571a659f6e1166321c5e"} Oct 31 03:31:29.492612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:29.492445 4488 scope.go:122] "RemoveContainer" containerID="efe25d3456b6e8dc2aa5c530ca14de386cf4b62538ad571a659f6e1166321c5e" Oct 31 03:31:29.493835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:29.493807 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:29.493897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:29.493875 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:29.493936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:29.493906 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:30.498254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:30.498224 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" event={"ID":"6a18583d-f061-46bb-8af6-9b34518d4d95","Type":"ContainerStarted","Data":"d5dd09bb5b27859a91076a55a1d2c2a4bff341216f1b4eddae02127c6fe18664"} Oct 31 03:31:31.093793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:31.093761 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:31:31.094012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:31.093817 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 03:31:31.094012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:31.093877 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 03:31:31.429825 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:31.429691 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:32.025224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:32.025186 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:32.093928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:32.093884 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 03:31:32.094156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:32.093951 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 03:31:32.252943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:32.252865 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:33.093804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:33.093764 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 03:31:33.094232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:33.093823 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 03:31:33.826907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:33.826688 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:34.094124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:34.094070 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 03:31:34.094526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:34.094179 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 03:31:35.094030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.093992 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 03:31:35.094273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.094056 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 03:31:35.816849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.816804 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj"] Oct 31 03:31:35.820206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.820181 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:35.922617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.922579 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"openshift-service-ca.crt\"" Oct 31 03:31:35.922878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.922860 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"kube-root-ca.crt\"" Oct 31 03:31:35.923057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.922924 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"default-dockercfg-lsfwl\"" Oct 31 03:31:35.923057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.922937 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"simulator-w1-e00ad3-kserve-self-signed-certs\"" Oct 31 03:31:35.924988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.924962 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj"] Oct 31 03:31:35.930874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.930849 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:35.942197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.942170 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvh7s\" (UniqueName: \"kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:35.942333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.942315 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:35.942374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.942356 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:35.942417 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.942384 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:35.942480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.942466 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:35.942521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:35.942499 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.043518 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.043471 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.043518 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.043515 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.043789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.043541 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.043789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.043574 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.043789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.043599 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.043789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.043625 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvh7s\" (UniqueName: \"kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.044051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.044027 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.044147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.044118 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.044207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.044156 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.046211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.046184 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.046718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.046698 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.097768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.097741 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:31:36.098271 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.098208 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:31:36.098996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.098977 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:31:36.105082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.105058 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvh7s\" (UniqueName: \"kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s\") pod \"simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.132170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.132127 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:36.287269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.287236 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:36.508877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.508847 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj"] Oct 31 03:31:36.512180 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:31:36.512149 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3d8d2f4_260a_46fc_81e2_f2f39b73597b.slice/crio-15f543af0b5b60154c3b826801ad415d194961be654503745a1fd19026ab199a WatchSource:0}: Error finding container 15f543af0b5b60154c3b826801ad415d194961be654503745a1fd19026ab199a: Status 404 returned error can't find the container with id 15f543af0b5b60154c3b826801ad415d194961be654503745a1fd19026ab199a Oct 31 03:31:36.546196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.546078 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" event={"ID":"c3d8d2f4-260a-46fc-81e2-f2f39b73597b","Type":"ContainerStarted","Data":"15f543af0b5b60154c3b826801ad415d194961be654503745a1fd19026ab199a"} Oct 31 03:31:36.591872 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:31:36.591831 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:31:36.893377 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:36.893343 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:37.053754 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.053722 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:37.554575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.554487 4488 generic.go:356] "Generic (PLEG): container finished" podID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerID="006a8ea9486d30db3754ca42dff98287bd0c835d4ef209be5582687a405a7179" exitCode=0 Oct 31 03:31:37.554575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.554551 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" event={"ID":"6eb50f54-f39c-4cac-bb0b-039339b0c778","Type":"ContainerDied","Data":"006a8ea9486d30db3754ca42dff98287bd0c835d4ef209be5582687a405a7179"} Oct 31 03:31:37.555168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.554961 4488 scope.go:122] "RemoveContainer" containerID="006a8ea9486d30db3754ca42dff98287bd0c835d4ef209be5582687a405a7179" Oct 31 03:31:37.556426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.556396 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:37.556546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.556461 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" event={"ID":"c3d8d2f4-260a-46fc-81e2-f2f39b73597b","Type":"ContainerStarted","Data":"7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd"} Oct 31 03:31:37.556546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.556484 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:37.556546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.556535 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:31:37.856139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:37.856066 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podStartSLOduration=2.856046901 podStartE2EDuration="2.856046901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:37.855008929 +0000 UTC m=+725.651727185" watchObservedRunningTime="2026-10-31 03:31:37.856046901 +0000 UTC m=+725.652765161" Oct 31 03:31:38.132344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:38.132286 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:38.563680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:38.563585 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" event={"ID":"6eb50f54-f39c-4cac-bb0b-039339b0c778","Type":"ContainerStarted","Data":"22bdc596bd77d3960f2d4114b68c3d6c4bfb690a27b1d47722160c4a9c3549a2"} Oct 31 03:31:38.779150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:38.779120 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:31:38.780084 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:38.780059 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:38.780226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:38.780131 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 03:31:39.018235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:39.018198 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:31:39.043769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:39.043726 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:39.779664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:39.779630 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:39.780137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:39.779695 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 03:31:40.779462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:40.779425 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:40.779653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:40.779480 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 03:31:41.779521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:41.779485 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:41.779970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:41.779552 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 03:31:42.212115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:42.211470 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:31:42.780109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:42.779601 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 31 03:31:42.780109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:42.779661 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 31 03:31:43.783391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:43.783361 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:31:43.784418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:43.784399 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:31:44.110579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:44.110546 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:46.132647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:46.132602 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:46.146034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:46.145997 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:46.614653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:46.614624 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:31:48.201031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:48.201002 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:48.804940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:48.804908 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:48.963606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:48.963578 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:49.118256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:49.118206 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:49.272860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:49.272830 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:52.054961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:52.054926 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:52.256210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:52.256176 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:52.665157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:52.665119 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:52.830491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:52.830449 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:54.625080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:54.625042 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:57.839419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:57.839385 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:58.499418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:58.499362 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:58.698795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:58.697920 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:59.731907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:31:59.731874 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:00.969703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:00.969664 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg"] Oct 31 03:32:00.973755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:00.973722 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.026701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.026654 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.027683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.027660 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5dh\" (UniqueName: \"kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.028016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.027998 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.028220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.028205 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.028360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.028347 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.028521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.028501 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.069693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.069655 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:01.084074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.084039 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"premium-w6-b63425-kserve-self-signed-certs\"" Oct 31 03:32:01.086213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.086187 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg"] Oct 31 03:32:01.129441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129409 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129460 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129624 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129675 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129751 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129815 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5dh\" (UniqueName: \"kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129887 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.129983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.129960 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.130156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.130133 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.132694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.132671 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.133057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.133035 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.191475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.191445 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5dh\" (UniqueName: \"kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh\") pod \"premium-w6-b63425-kserve-56dc655b6-l7kkg\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.286541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.286448 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:01.680819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:01.680793 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg"] Oct 31 03:32:01.682912 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:32:01.682883 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf468edbd_35ea_4024_8dcd_36386b294e5a.slice/crio-d6702612a16eebd7f020284cee3530675a3805c2a08709d0eeeb86d034858792 WatchSource:0}: Error finding container d6702612a16eebd7f020284cee3530675a3805c2a08709d0eeeb86d034858792: Status 404 returned error can't find the container with id d6702612a16eebd7f020284cee3530675a3805c2a08709d0eeeb86d034858792 Oct 31 03:32:02.681870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:02.681828 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" event={"ID":"f468edbd-35ea-4024-8dcd-36386b294e5a","Type":"ContainerStarted","Data":"0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea"} Oct 31 03:32:02.681870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:02.681878 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" event={"ID":"f468edbd-35ea-4024-8dcd-36386b294e5a","Type":"ContainerStarted","Data":"d6702612a16eebd7f020284cee3530675a3805c2a08709d0eeeb86d034858792"} Oct 31 03:32:02.800884 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:02.800831 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podStartSLOduration=2.800816012 podStartE2EDuration="2.800816012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:02.799193148 +0000 UTC m=+750.595911408" watchObservedRunningTime="2026-10-31 03:32:02.800816012 +0000 UTC m=+750.597534271" Oct 31 03:32:03.287293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:03.287250 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:04.600251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:04.600216 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:06.589798 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:32:06.589763 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:32:06.771660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:06.771621 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:32:07.443049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:07.443007 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:08.246164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:08.246129 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:08.879131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:08.879057 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.041683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:09.041644 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.910534 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:09.910499 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:11.287028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:11.286987 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:11.301743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:11.301693 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:11.741782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:11.741746 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:32:12.637115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.637054 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7"] Oct 31 03:32:12.640200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.640181 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.693187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.693159 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"kube-root-ca.crt\"" Oct 31 03:32:12.733727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.733687 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:12.742583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.742550 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"default-dockercfg-s5hcs\"" Oct 31 03:32:12.742583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.742568 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"openshift-service-ca.crt\"" Oct 31 03:32:12.742583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.742578 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"simulator-w0-909365-kserve-self-signed-certs\"" Oct 31 03:32:12.744990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.744970 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7"] Oct 31 03:32:12.760366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.760335 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.760547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.760381 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.760547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.760407 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.760547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.760487 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.760650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.760583 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.760800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.760776 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h549l\" (UniqueName: \"kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.861882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.861826 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.861882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.861893 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.861929 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.861955 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.862247 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h549l\" (UniqueName: \"kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.862351 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.862368 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.862356 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.862551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.862468 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.864463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.864440 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.864829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.864812 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.924118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.924007 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h549l\" (UniqueName: \"kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l\") pod \"simulator-w0-909365-kserve-58648bbd79-xbvz7\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:12.951958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:12.951920 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:13.329630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:13.329584 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7"] Oct 31 03:32:13.331993 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:32:13.331970 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219920c9_66f5_48e8_87af_011113d40627.slice/crio-9a17c22f2c536bf7239929c749d11d14ed3f748dacb6507ed19d385c4908c6ea WatchSource:0}: Error finding container 9a17c22f2c536bf7239929c749d11d14ed3f748dacb6507ed19d385c4908c6ea: Status 404 returned error can't find the container with id 9a17c22f2c536bf7239929c749d11d14ed3f748dacb6507ed19d385c4908c6ea Oct 31 03:32:13.742974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:13.742879 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" event={"ID":"219920c9-66f5-48e8-87af-011113d40627","Type":"ContainerStarted","Data":"08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e"} Oct 31 03:32:13.742974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:13.742925 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" event={"ID":"219920c9-66f5-48e8-87af-011113d40627","Type":"ContainerStarted","Data":"9a17c22f2c536bf7239929c749d11d14ed3f748dacb6507ed19d385c4908c6ea"} Oct 31 03:32:13.862640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:13.862535 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podStartSLOduration=1.862516819 podStartE2EDuration="1.862516819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:13.860614954 +0000 UTC m=+761.657333239" watchObservedRunningTime="2026-10-31 03:32:13.862516819 +0000 UTC m=+761.659235144" Oct 31 03:32:13.952893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:13.952848 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:14.715620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:14.715573 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:19.519723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:19.519681 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:20.522994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:20.522952 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:22.952306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:22.952262 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:22.965419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:22.965393 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:23.810252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:23.810222 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:32:24.409042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:24.409007 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:25.865981 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:25.865948 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:29.217273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:29.217240 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.037036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:32.036996 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:32:33.483429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:33.483390 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:34.014343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:34.014308 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:36.589028 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:32:36.588991 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:32:38.897780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:38.897745 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:43.701241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:43.701201 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:48.617195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:48.617161 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.457226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:52.457177 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:53.420614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:53.420578 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:54.626399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:54.626355 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:32:57.028974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.028940 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk"] Oct 31 03:32:57.031447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.031429 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.133737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.133706 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"premium-w1-e00ad3-kserve-self-signed-certs\"" Oct 31 03:32:57.134609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.134579 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:57.135919 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.135886 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk"] Oct 31 03:32:57.156696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.156661 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.156696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.156700 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgwt\" (UniqueName: \"kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.156908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.156855 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.156960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.156939 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.157017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.157005 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.157053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.157039 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258352 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258416 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258458 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258493 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258519 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258547 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgwt\" (UniqueName: \"kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.258960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.258937 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.259041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.259003 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.259041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.259019 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.261074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.261050 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.261227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.261209 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.320209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.320139 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgwt\" (UniqueName: \"kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt\") pod \"premium-w1-e00ad3-kserve-6d495cb544-rpsmk\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.343723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.343680 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:32:57.726863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.726831 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk"] Oct 31 03:32:57.729120 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:32:57.729057 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1475d24_0c69_48d0_a7d3_8357456e1e36.slice/crio-75191c2697e72259c89a296c7ede5f190de3f5ca3b67799a4b4b1aeec49a1b3e WatchSource:0}: Error finding container 75191c2697e72259c89a296c7ede5f190de3f5ca3b67799a4b4b1aeec49a1b3e: Status 404 returned error can't find the container with id 75191c2697e72259c89a296c7ede5f190de3f5ca3b67799a4b4b1aeec49a1b3e Oct 31 03:32:57.993013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.992919 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" event={"ID":"c1475d24-0c69-48d0-a7d3-8357456e1e36","Type":"ContainerStarted","Data":"c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937"} Oct 31 03:32:57.993013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:57.992961 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" event={"ID":"c1475d24-0c69-48d0-a7d3-8357456e1e36","Type":"ContainerStarted","Data":"75191c2697e72259c89a296c7ede5f190de3f5ca3b67799a4b4b1aeec49a1b3e"} Oct 31 03:32:58.110794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:58.110739 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podStartSLOduration=2.110723996 podStartE2EDuration="2.110723996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:58.109697619 +0000 UTC m=+805.906415898" watchObservedRunningTime="2026-10-31 03:32:58.110723996 +0000 UTC m=+805.907442255" Oct 31 03:32:58.293140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:58.293028 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:58.344431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:32:58.344382 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:33:03.101367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:03.101335 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:06.591442 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:33:06.591399 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:33:07.343890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:07.343856 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:33:07.356788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:07.356754 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:33:07.900413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:07.900382 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:08.056246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:08.056221 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:33:12.798442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:12.798408 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:15.812965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.812933 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf"] Oct 31 03:33:15.815538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.815523 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:15.873497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.873460 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"kube-root-ca.crt\"" Oct 31 03:33:15.917995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.917950 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:15.918273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.918257 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"openshift-service-ca.crt\"" Oct 31 03:33:15.918363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.918275 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"premium-w3-b98893-kserve-self-signed-certs\"" Oct 31 03:33:15.918363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.918275 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b98893\"/\"default-dockercfg-wznxn\"" Oct 31 03:33:15.920355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.920331 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf"] Oct 31 03:33:15.959815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.959775 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:15.960225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.960203 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:15.960400 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.960379 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmk5\" (UniqueName: \"kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:15.960659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.960632 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:15.960781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.960675 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:15.960895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:15.960876 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.062384 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062341 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmk5\" (UniqueName: \"kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.062575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062391 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.062575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062411 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.062575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062461 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.062575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062497 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.062575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062564 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.063027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062962 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.063027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.062991 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.063220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.063028 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.064873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.064854 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.065212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.065194 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.124627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.124588 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmk5\" (UniqueName: \"kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5\") pod \"premium-w3-b98893-kserve-57648f8784-4qqlf\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.127436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.127400 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:16.517942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:16.517756 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf"] Oct 31 03:33:16.520322 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:33:16.520296 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5ef745_c36e_4038_b875_7940030f51d2.slice/crio-efacb776976fca35f011f0176a7b2951ba0cd305eac47323f4d1097dbfef7d46 WatchSource:0}: Error finding container efacb776976fca35f011f0176a7b2951ba0cd305eac47323f4d1097dbfef7d46: Status 404 returned error can't find the container with id efacb776976fca35f011f0176a7b2951ba0cd305eac47323f4d1097dbfef7d46 Oct 31 03:33:17.093197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:17.093157 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" event={"ID":"fb5ef745-c36e-4038-b875-7940030f51d2","Type":"ContainerStarted","Data":"4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5"} Oct 31 03:33:17.093612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:17.093215 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" event={"ID":"fb5ef745-c36e-4038-b875-7940030f51d2","Type":"ContainerStarted","Data":"efacb776976fca35f011f0176a7b2951ba0cd305eac47323f4d1097dbfef7d46"} Oct 31 03:33:17.128475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:17.128444 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:17.210639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:17.210575 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podStartSLOduration=2.210557753 podStartE2EDuration="2.210557753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:17.210199171 +0000 UTC m=+825.006917432" watchObservedRunningTime="2026-10-31 03:33:17.210557753 +0000 UTC m=+825.007276013" Oct 31 03:33:17.715332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:17.715295 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:20.858731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:20.858692 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:21.941333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:21.941291 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:33:22.527910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:22.527858 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:26.128662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:26.128615 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:26.144527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:26.144499 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:26.156378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:26.156354 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:33:27.328700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:27.328665 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:32.270843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:32.270804 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:36.591545 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:33:36.591496 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:33:36.998492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:36.998407 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:40.566854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.566810 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv"] Oct 31 03:33:40.570788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.570772 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.673298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.673263 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"premium-w0-909365-kserve-self-signed-certs\"" Oct 31 03:33:40.674641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.674609 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:40.675742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.675718 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv"] Oct 31 03:33:40.746059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.746029 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.746278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.746102 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.746278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.746202 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.746278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.746248 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcc8t\" (UniqueName: \"kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.746408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.746312 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.746408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.746357 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.847460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847425 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.847678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847478 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.847678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847514 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcc8t\" (UniqueName: \"kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.847678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847545 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.847678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847564 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.847678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847658 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.848002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847973 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.848002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.847998 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.848191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.848041 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.849952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.849927 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.850117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.850078 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:40.908840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:40.908809 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcc8t\" (UniqueName: \"kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t\") pod \"premium-w0-909365-kserve-74b7d98cc8-72gfv\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:41.207170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:41.207035 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:41.584715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:41.584686 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv"] Oct 31 03:33:41.587170 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:33:41.587139 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65ee8d7e_e337_4970_862f_9da26774ccbb.slice/crio-79d1a2725cc5fc2c602b9266eb5a3b52ebe4caf96d2263e050eb2c927fc45c11 WatchSource:0}: Error finding container 79d1a2725cc5fc2c602b9266eb5a3b52ebe4caf96d2263e050eb2c927fc45c11: Status 404 returned error can't find the container with id 79d1a2725cc5fc2c602b9266eb5a3b52ebe4caf96d2263e050eb2c927fc45c11 Oct 31 03:33:41.813216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:41.813180 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:42.069949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.069857 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng"] Oct 31 03:33:42.073605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.073588 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.125975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.125942 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:33:42.125975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.125961 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:33:42.165393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.165342 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sv4\" (UniqueName: \"kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.165613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.165494 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.165613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.165534 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.176039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.176002 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:33:42.176226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.176147 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:33:42.178251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.178229 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng"] Oct 31 03:33:42.242585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.242549 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" event={"ID":"65ee8d7e-e337-4970-862f-9da26774ccbb","Type":"ContainerStarted","Data":"e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735"} Oct 31 03:33:42.242771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.242599 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" event={"ID":"65ee8d7e-e337-4970-862f-9da26774ccbb","Type":"ContainerStarted","Data":"79d1a2725cc5fc2c602b9266eb5a3b52ebe4caf96d2263e050eb2c927fc45c11"} Oct 31 03:33:42.266410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.266371 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.266600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.266427 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.266600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.266483 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sv4\" (UniqueName: \"kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.267146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.267127 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.268850 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.268830 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.328021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.327998 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sv4\" (UniqueName: \"kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4\") pod \"maas-api-revoke-sub-11fb2b05-l87ng\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.361248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.361193 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podStartSLOduration=2.361179022 podStartE2EDuration="2.361179022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:42.359307455 +0000 UTC m=+850.156025717" watchObservedRunningTime="2026-10-31 03:33:42.361179022 +0000 UTC m=+850.157897282" Oct 31 03:33:42.386844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.386810 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:42.764799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.764774 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng"] Oct 31 03:33:42.767798 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:33:42.767770 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806d6f58_965d_42c4_b62c_80d51fd523f7.slice/crio-16691168b64d5c3cc181bec4a75bd0a12542241338e079e63f66706c7f69e00a WatchSource:0}: Error finding container 16691168b64d5c3cc181bec4a75bd0a12542241338e079e63f66706c7f69e00a: Status 404 returned error can't find the container with id 16691168b64d5c3cc181bec4a75bd0a12542241338e079e63f66706c7f69e00a Oct 31 03:33:42.770700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:42.770610 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:33:43.207966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:43.207919 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:43.249047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:43.249013 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" event={"ID":"806d6f58-965d-42c4-b62c-80d51fd523f7","Type":"ContainerStarted","Data":"16691168b64d5c3cc181bec4a75bd0a12542241338e079e63f66706c7f69e00a"} Oct 31 03:33:46.616426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:46.616390 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:47.274612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:47.274580 4488 generic.go:356] "Generic (PLEG): container finished" podID="806d6f58-965d-42c4-b62c-80d51fd523f7" containerID="700475cab5a7fe19017743c98966e29208e329a33e12041fe80e130672bcd4bc" exitCode=0 Oct 31 03:33:47.274812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:47.274660 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" event={"ID":"806d6f58-965d-42c4-b62c-80d51fd523f7","Type":"ContainerDied","Data":"700475cab5a7fe19017743c98966e29208e329a33e12041fe80e130672bcd4bc"} Oct 31 03:33:47.982755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:47.982716 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:33:48.430270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.430244 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:48.524308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.524277 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4\" (UniqueName: \"kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4\") pod \"806d6f58-965d-42c4-b62c-80d51fd523f7\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " Oct 31 03:33:48.524479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.524371 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca\") pod \"806d6f58-965d-42c4-b62c-80d51fd523f7\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " Oct 31 03:33:48.524520 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.524470 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca\") pod \"806d6f58-965d-42c4-b62c-80d51fd523f7\" (UID: \"806d6f58-965d-42c4-b62c-80d51fd523f7\") " Oct 31 03:33:48.524821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.524790 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca" pod "806d6f58-965d-42c4-b62c-80d51fd523f7" (UID: "806d6f58-965d-42c4-b62c-80d51fd523f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:48.526874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.526853 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca" pod "806d6f58-965d-42c4-b62c-80d51fd523f7" (UID: "806d6f58-965d-42c4-b62c-80d51fd523f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:33:48.526965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.526947 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4" pod "806d6f58-965d-42c4-b62c-80d51fd523f7" (UID: "806d6f58-965d-42c4-b62c-80d51fd523f7"). InnerVolumeSpecName "kube-api-access-86sv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:48.626683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.626642 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:33:48.626683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.626685 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d6f58-965d-42c4-b62c-80d51fd523f7-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:33:48.626953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:48.626703 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86sv4\" (UniqueName: \"kubernetes.io/projected/806d6f58-965d-42c4-b62c-80d51fd523f7-kube-api-access-86sv4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:33:49.287380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:49.287347 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" Oct 31 03:33:49.287867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:49.287381 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng" event={"ID":"806d6f58-965d-42c4-b62c-80d51fd523f7","Type":"ContainerDied","Data":"16691168b64d5c3cc181bec4a75bd0a12542241338e079e63f66706c7f69e00a"} Oct 31 03:33:49.287867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:49.287427 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16691168b64d5c3cc181bec4a75bd0a12542241338e079e63f66706c7f69e00a" Oct 31 03:33:51.139022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:51.138979 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:51.208150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:51.208123 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:51.221758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:51.221727 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:51.309568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:51.309542 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:33:51.523676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:51.523597 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:56.522693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:56.522658 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:57.489655 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.489610 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5"] Oct 31 03:33:57.490156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.490137 4488 cpu_manager.go:414] "containerMap: removing container" podUID="806d6f58-965d-42c4-b62c-80d51fd523f7" containerName="revoke-keys" Oct 31 03:33:57.490268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.490158 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806d6f58-965d-42c4-b62c-80d51fd523f7" containerName="revoke-keys" Oct 31 03:33:57.490336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.490320 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="806d6f58-965d-42c4-b62c-80d51fd523f7" containerName="revoke-keys" Oct 31 03:33:57.493795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.493773 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.548758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.548722 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:33:57.598621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.598592 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:33:57.598621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.598605 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:33:57.598835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.598701 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:33:57.600771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.600753 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5"] Oct 31 03:33:57.624430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.624402 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.624602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.624474 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.624602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.624562 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl8h\" (UniqueName: \"kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.725855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.725817 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.726064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.725902 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.726064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.725934 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl8h\" (UniqueName: \"kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.726519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.726497 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.728482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.728463 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.787572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.787489 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl8h\" (UniqueName: \"kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h\") pod \"maas-api-revoke-sub-246644c7-q5wd5\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:57.806917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:57.806884 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:33:58.183880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.183854 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5"] Oct 31 03:33:58.186176 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:33:58.186148 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d2b383_7f47_42a0_af4a_b7daf1d37e1d.slice/crio-89c00cfc2f5381d7be55af7fc6cd4692c227ea4727c44c29f25937be49a41e94 WatchSource:0}: Error finding container 89c00cfc2f5381d7be55af7fc6cd4692c227ea4727c44c29f25937be49a41e94: Status 404 returned error can't find the container with id 89c00cfc2f5381d7be55af7fc6cd4692c227ea4727c44c29f25937be49a41e94 Oct 31 03:33:58.203653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.203625 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt"] Oct 31 03:33:58.208640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.208618 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.264065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.264035 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"kube-root-ca.crt\"" Oct 31 03:33:58.293062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.293034 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"openshift-service-ca.crt\"" Oct 31 03:33:58.293200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.293073 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"premium-w5-14cb5a-kserve-self-signed-certs\"" Oct 31 03:33:58.293200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.293183 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"default-dockercfg-gncl9\"" Oct 31 03:33:58.306460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.306205 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:58.316251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.316225 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt"] Oct 31 03:33:58.333424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.333396 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.333579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.333536 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.333618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.333582 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.333618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.333599 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.333718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.333704 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.333751 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.333743 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdc2\" (UniqueName: \"kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.335483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.335456 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" event={"ID":"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d","Type":"ContainerStarted","Data":"89c00cfc2f5381d7be55af7fc6cd4692c227ea4727c44c29f25937be49a41e94"} Oct 31 03:33:58.434960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.434888 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.434960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.434923 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.434969 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.434996 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdc2\" (UniqueName: \"kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.435115 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.435195 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.435443 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.435482 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.435599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.435582 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.437399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.437381 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.437651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.437636 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.496292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.496260 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdc2\" (UniqueName: \"kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2\") pod \"premium-w5-14cb5a-kserve-698496f87-7dwkt\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.537245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.537217 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:58.935558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:58.935527 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt"] Oct 31 03:33:58.937780 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:33:58.937750 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c55bc84_c08a_418e_b19a_b8b97fa4bab6.slice/crio-60fbabb1eda5f2375173b124dc18979359fdc0f96919dbd70bbb362fc0ad6fd9 WatchSource:0}: Error finding container 60fbabb1eda5f2375173b124dc18979359fdc0f96919dbd70bbb362fc0ad6fd9: Status 404 returned error can't find the container with id 60fbabb1eda5f2375173b124dc18979359fdc0f96919dbd70bbb362fc0ad6fd9 Oct 31 03:33:59.342743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:59.342709 4488 generic.go:356] "Generic (PLEG): container finished" podID="f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" containerID="1e271b1d0304af046fa3067bf03596b00227ce65f907732e54362466a3bb3abc" exitCode=0 Oct 31 03:33:59.342953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:59.342791 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" event={"ID":"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d","Type":"ContainerDied","Data":"1e271b1d0304af046fa3067bf03596b00227ce65f907732e54362466a3bb3abc"} Oct 31 03:33:59.350491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:59.350449 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" event={"ID":"9c55bc84-c08a-418e-b19a-b8b97fa4bab6","Type":"ContainerStarted","Data":"ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3"} Oct 31 03:33:59.350681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:59.350508 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" event={"ID":"9c55bc84-c08a-418e-b19a-b8b97fa4bab6","Type":"ContainerStarted","Data":"60fbabb1eda5f2375173b124dc18979359fdc0f96919dbd70bbb362fc0ad6fd9"} Oct 31 03:33:59.537826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:59.537789 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:33:59.578851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:33:59.578796 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podStartSLOduration=1.57878278 podStartE2EDuration="1.57878278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:59.576288695 +0000 UTC m=+867.373006953" watchObservedRunningTime="2026-10-31 03:33:59.57878278 +0000 UTC m=+867.375501037" Oct 31 03:34:00.516904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.516881 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:34:00.558729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.558696 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca\") pod \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " Oct 31 03:34:00.558729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.558741 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h\" (UniqueName: \"kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h\") pod \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " Oct 31 03:34:00.558979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.558865 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca\") pod \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\" (UID: \"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d\") " Oct 31 03:34:00.559308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.559275 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca" pod "f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" (UID: "f9d2b383-7f47-42a0-af4a-b7daf1d37e1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:00.559445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.559393 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:00.561023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.561001 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca" pod "f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" (UID: "f9d2b383-7f47-42a0-af4a-b7daf1d37e1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:34:00.561268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.561247 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h" pod "f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" (UID: "f9d2b383-7f47-42a0-af4a-b7daf1d37e1d"). InnerVolumeSpecName "kube-api-access-brl8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:00.660820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.660727 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:00.660820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:00.660759 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brl8h\" (UniqueName: \"kubernetes.io/projected/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d-kube-api-access-brl8h\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:01.368017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:01.367944 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" Oct 31 03:34:01.377325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:01.377287 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5" event={"ID":"f9d2b383-7f47-42a0-af4a-b7daf1d37e1d","Type":"ContainerDied","Data":"89c00cfc2f5381d7be55af7fc6cd4692c227ea4727c44c29f25937be49a41e94"} Oct 31 03:34:01.377543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:01.377361 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c00cfc2f5381d7be55af7fc6cd4692c227ea4727c44c29f25937be49a41e94" Oct 31 03:34:01.535328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:01.535273 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:06.425779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:06.425726 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:06.588817 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:34:06.588780 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:34:06.785031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:06.784918 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:06.942784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:06.942745 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:08.537657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:08.537615 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:34:08.550371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:08.550345 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:34:09.424560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:09.424530 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:34:10.461145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:10.461109 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:11.653181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:11.653149 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:13.484189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.484147 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs"] Oct 31 03:34:13.484776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.484755 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" containerName="revoke-keys" Oct 31 03:34:13.484831 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.484779 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" containerName="revoke-keys" Oct 31 03:34:13.484867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.484844 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" containerName="revoke-keys" Oct 31 03:34:13.488165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.488147 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.593665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.593634 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:34:13.593665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.593642 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:34:13.593886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.593687 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:34:13.593886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.593702 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:34:13.595949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.595913 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs"] Oct 31 03:34:13.621872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.621837 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpvc\" (UniqueName: \"kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.622053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.621880 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.622053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.622026 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.723247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.723213 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpvc\" (UniqueName: \"kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.723445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.723256 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.723445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.723329 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.723955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.723911 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.725858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.725837 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.785162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.785057 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpvc\" (UniqueName: \"kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc\") pod \"maas-api-revoke-sub-d4420ac5-8qpcs\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:13.801913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:13.801873 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:14.203199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:14.203175 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs"] Oct 31 03:34:14.204954 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:34:14.204922 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e8159e_7548_417e_b5a9_81373985ea62.slice/crio-48ef8e86d1db93d9aecd66dd4a12ed21d162cf7951da9b977bd2323543b8aed2 WatchSource:0}: Error finding container 48ef8e86d1db93d9aecd66dd4a12ed21d162cf7951da9b977bd2323543b8aed2: Status 404 returned error can't find the container with id 48ef8e86d1db93d9aecd66dd4a12ed21d162cf7951da9b977bd2323543b8aed2 Oct 31 03:34:14.439924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:14.439883 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" event={"ID":"d6e8159e-7548-417e-b5a9-81373985ea62","Type":"ContainerStarted","Data":"48ef8e86d1db93d9aecd66dd4a12ed21d162cf7951da9b977bd2323543b8aed2"} Oct 31 03:34:15.446925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:15.446888 4488 generic.go:356] "Generic (PLEG): container finished" podID="d6e8159e-7548-417e-b5a9-81373985ea62" containerID="fd4479d12adb47dbe034b2737d2cd6f3d4b3fb2fc9cb9a6a9a4078a40b1a2b3e" exitCode=0 Oct 31 03:34:15.447343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:15.446973 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" event={"ID":"d6e8159e-7548-417e-b5a9-81373985ea62","Type":"ContainerDied","Data":"fd4479d12adb47dbe034b2737d2cd6f3d4b3fb2fc9cb9a6a9a4078a40b1a2b3e"} Oct 31 03:34:15.810799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:15.810709 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:34:16.643118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.642627 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:16.701686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.701653 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:16.858683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.858641 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca\") pod \"d6e8159e-7548-417e-b5a9-81373985ea62\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " Oct 31 03:34:16.858910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.858801 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca\") pod \"d6e8159e-7548-417e-b5a9-81373985ea62\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " Oct 31 03:34:16.858995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.858927 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc\" (UniqueName: \"kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc\") pod \"d6e8159e-7548-417e-b5a9-81373985ea62\" (UID: \"d6e8159e-7548-417e-b5a9-81373985ea62\") " Oct 31 03:34:16.859704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.859671 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca" pod "d6e8159e-7548-417e-b5a9-81373985ea62" (UID: "d6e8159e-7548-417e-b5a9-81373985ea62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:16.872420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.872338 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca" pod "d6e8159e-7548-417e-b5a9-81373985ea62" (UID: "d6e8159e-7548-417e-b5a9-81373985ea62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:34:16.874756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.874720 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc" pod "d6e8159e-7548-417e-b5a9-81373985ea62" (UID: "d6e8159e-7548-417e-b5a9-81373985ea62"). InnerVolumeSpecName "kube-api-access-tcpvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:16.961403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.961360 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:16.961668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.961649 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e8159e-7548-417e-b5a9-81373985ea62-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:16.961788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:16.961774 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcpvc\" (UniqueName: \"kubernetes.io/projected/d6e8159e-7548-417e-b5a9-81373985ea62-kube-api-access-tcpvc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:17.462341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:17.462302 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" event={"ID":"d6e8159e-7548-417e-b5a9-81373985ea62","Type":"ContainerDied","Data":"48ef8e86d1db93d9aecd66dd4a12ed21d162cf7951da9b977bd2323543b8aed2"} Oct 31 03:34:17.462341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:17.462335 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs" Oct 31 03:34:17.462610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:17.462361 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ef8e86d1db93d9aecd66dd4a12ed21d162cf7951da9b977bd2323543b8aed2" Oct 31 03:34:21.216196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:21.216147 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:22.393514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:22.393471 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:26.679310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:26.679208 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:31.612269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:31.612232 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:33.706871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.706831 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs"] Oct 31 03:34:33.707515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.707493 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d6e8159e-7548-417e-b5a9-81373985ea62" containerName="revoke-keys" Oct 31 03:34:33.707609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.707518 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e8159e-7548-417e-b5a9-81373985ea62" containerName="revoke-keys" Oct 31 03:34:33.707685 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.707672 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e8159e-7548-417e-b5a9-81373985ea62" containerName="revoke-keys" Oct 31 03:34:33.714668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.714643 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.767689 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.767653 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:34:33.768210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.768178 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:34:33.768343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.768258 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:34:33.769399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.769361 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs"] Oct 31 03:34:33.816055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.816022 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:34:33.858072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.858033 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.858294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.858120 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxsc\" (UniqueName: \"kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.858358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.858329 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.959677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.959582 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.959677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.959654 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.959677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.959678 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxsc\" (UniqueName: \"kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.960310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.960289 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:33.962234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:33.962209 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:34.021431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:34.021396 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxsc\" (UniqueName: \"kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc\") pod \"maas-api-revoke-sub-62530746-r4kbs\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:34.028167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:34.028140 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:34.403802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:34.403778 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs"] Oct 31 03:34:34.405717 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:34:34.405687 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeceec002_bf44_414f_95e3_be7eb6c20d2b.slice/crio-8c805057377eb0e58baee4916017c7ea4da50bda82d449803368314f7b232022 WatchSource:0}: Error finding container 8c805057377eb0e58baee4916017c7ea4da50bda82d449803368314f7b232022: Status 404 returned error can't find the container with id 8c805057377eb0e58baee4916017c7ea4da50bda82d449803368314f7b232022 Oct 31 03:34:34.555254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:34.555223 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" event={"ID":"eceec002-bf44-414f-95e3-be7eb6c20d2b","Type":"ContainerStarted","Data":"8c805057377eb0e58baee4916017c7ea4da50bda82d449803368314f7b232022"} Oct 31 03:34:35.318471 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:34:35.318438 4488 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeceec002_bf44_414f_95e3_be7eb6c20d2b.slice/crio-d67e6db46e6e506d81837aee4803cf4dee5e4c8bdda3ae56c3388f463e282e21.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:34:35.565622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:35.565514 4488 generic.go:356] "Generic (PLEG): container finished" podID="eceec002-bf44-414f-95e3-be7eb6c20d2b" containerID="d67e6db46e6e506d81837aee4803cf4dee5e4c8bdda3ae56c3388f463e282e21" exitCode=0 Oct 31 03:34:35.565622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:35.565596 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" event={"ID":"eceec002-bf44-414f-95e3-be7eb6c20d2b","Type":"ContainerDied","Data":"d67e6db46e6e506d81837aee4803cf4dee5e4c8bdda3ae56c3388f463e282e21"} Oct 31 03:34:36.600707 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:34:36.600669 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:34:36.720044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.720015 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:36.742079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.742044 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:36.898548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.898443 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca\") pod \"eceec002-bf44-414f-95e3-be7eb6c20d2b\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " Oct 31 03:34:36.898548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.898482 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc\" (UniqueName: \"kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc\") pod \"eceec002-bf44-414f-95e3-be7eb6c20d2b\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " Oct 31 03:34:36.898785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.898604 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca\") pod \"eceec002-bf44-414f-95e3-be7eb6c20d2b\" (UID: \"eceec002-bf44-414f-95e3-be7eb6c20d2b\") " Oct 31 03:34:36.899325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.899272 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca" pod "eceec002-bf44-414f-95e3-be7eb6c20d2b" (UID: "eceec002-bf44-414f-95e3-be7eb6c20d2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:36.901681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.901646 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc" pod "eceec002-bf44-414f-95e3-be7eb6c20d2b" (UID: "eceec002-bf44-414f-95e3-be7eb6c20d2b"). InnerVolumeSpecName "kube-api-access-zqxsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:36.902230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:36.902200 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca" pod "eceec002-bf44-414f-95e3-be7eb6c20d2b" (UID: "eceec002-bf44-414f-95e3-be7eb6c20d2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:34:37.000189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.000143 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:37.000189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.000184 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceec002-bf44-414f-95e3-be7eb6c20d2b-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:37.000189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.000199 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxsc\" (UniqueName: \"kubernetes.io/projected/eceec002-bf44-414f-95e3-be7eb6c20d2b-kube-api-access-zqxsc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:37.246415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.246319 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64"] Oct 31 03:34:37.246755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.246739 4488 cpu_manager.go:414] "containerMap: removing container" podUID="eceec002-bf44-414f-95e3-be7eb6c20d2b" containerName="revoke-keys" Oct 31 03:34:37.246755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.246754 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eceec002-bf44-414f-95e3-be7eb6c20d2b" containerName="revoke-keys" Oct 31 03:34:37.246926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.246842 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="eceec002-bf44-414f-95e3-be7eb6c20d2b" containerName="revoke-keys" Oct 31 03:34:37.249949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.249926 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.309333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.309303 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64"] Oct 31 03:34:37.403203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.403168 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.403396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.403354 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gglwr\" (UniqueName: \"kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.403459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.403402 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.505040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.504951 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.505231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.505058 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglwr\" (UniqueName: \"kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.505231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.505114 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.505643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.505615 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.507590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.507574 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.569800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.569768 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gglwr\" (UniqueName: \"kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr\") pod \"maas-api-revoke-sub-795c5e7a-bjc64\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:37.583322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.583284 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" event={"ID":"eceec002-bf44-414f-95e3-be7eb6c20d2b","Type":"ContainerDied","Data":"8c805057377eb0e58baee4916017c7ea4da50bda82d449803368314f7b232022"} Oct 31 03:34:37.583494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.583337 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c805057377eb0e58baee4916017c7ea4da50bda82d449803368314f7b232022" Oct 31 03:34:37.583494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.583295 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs" Oct 31 03:34:37.864279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:37.864241 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:38.242309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:38.241944 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64"] Oct 31 03:34:38.245429 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:34:38.245397 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167d5711_a368_4260_a52d_07fc3f38b0e9.slice/crio-a0e2c503f8297265f63af87855e188517d0a95254446193fb4310233edaf463a WatchSource:0}: Error finding container a0e2c503f8297265f63af87855e188517d0a95254446193fb4310233edaf463a: Status 404 returned error can't find the container with id a0e2c503f8297265f63af87855e188517d0a95254446193fb4310233edaf463a Oct 31 03:34:38.590170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:38.590129 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" event={"ID":"167d5711-a368-4260-a52d-07fc3f38b0e9","Type":"ContainerStarted","Data":"06e8abebe26d66e839e48bfb85bbe20407477c0b16a49f97879ed1c1d9bc8069"} Oct 31 03:34:38.590374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:38.590184 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" event={"ID":"167d5711-a368-4260-a52d-07fc3f38b0e9","Type":"ContainerStarted","Data":"a0e2c503f8297265f63af87855e188517d0a95254446193fb4310233edaf463a"} Oct 31 03:34:38.707141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:38.707053 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" podStartSLOduration=1.707033805 podStartE2EDuration="1.707033805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:38.705764602 +0000 UTC m=+906.502482860" watchObservedRunningTime="2026-10-31 03:34:38.707033805 +0000 UTC m=+906.503752145" Oct 31 03:34:39.610676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:39.610639 4488 generic.go:356] "Generic (PLEG): container finished" podID="167d5711-a368-4260-a52d-07fc3f38b0e9" containerID="06e8abebe26d66e839e48bfb85bbe20407477c0b16a49f97879ed1c1d9bc8069" exitCode=0 Oct 31 03:34:39.611302 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:39.611272 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" event={"ID":"167d5711-a368-4260-a52d-07fc3f38b0e9","Type":"ContainerDied","Data":"06e8abebe26d66e839e48bfb85bbe20407477c0b16a49f97879ed1c1d9bc8069"} Oct 31 03:34:40.954878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:40.954843 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:41.052424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.052384 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca\") pod \"167d5711-a368-4260-a52d-07fc3f38b0e9\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " Oct 31 03:34:41.052649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.052528 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr\" (UniqueName: \"kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr\") pod \"167d5711-a368-4260-a52d-07fc3f38b0e9\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " Oct 31 03:34:41.052649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.052635 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca\") pod \"167d5711-a368-4260-a52d-07fc3f38b0e9\" (UID: \"167d5711-a368-4260-a52d-07fc3f38b0e9\") " Oct 31 03:34:41.053224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.053187 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca" pod "167d5711-a368-4260-a52d-07fc3f38b0e9" (UID: "167d5711-a368-4260-a52d-07fc3f38b0e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:41.058134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.057198 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca" pod "167d5711-a368-4260-a52d-07fc3f38b0e9" (UID: "167d5711-a368-4260-a52d-07fc3f38b0e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:34:41.058134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.057226 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr" pod "167d5711-a368-4260-a52d-07fc3f38b0e9" (UID: "167d5711-a368-4260-a52d-07fc3f38b0e9"). InnerVolumeSpecName "kube-api-access-gglwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:41.154752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.154717 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:41.154752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.154751 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gglwr\" (UniqueName: \"kubernetes.io/projected/167d5711-a368-4260-a52d-07fc3f38b0e9-kube-api-access-gglwr\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:41.154752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.154765 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d5711-a368-4260-a52d-07fc3f38b0e9-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:41.506621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.506587 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:41.629782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.629744 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" event={"ID":"167d5711-a368-4260-a52d-07fc3f38b0e9","Type":"ContainerDied","Data":"a0e2c503f8297265f63af87855e188517d0a95254446193fb4310233edaf463a"} Oct 31 03:34:41.629782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.629782 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64" Oct 31 03:34:41.630031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:41.629797 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e2c503f8297265f63af87855e188517d0a95254446193fb4310233edaf463a" Oct 31 03:34:44.551460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.551424 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d"] Oct 31 03:34:44.551951 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.551853 4488 cpu_manager.go:414] "containerMap: removing container" podUID="167d5711-a368-4260-a52d-07fc3f38b0e9" containerName="revoke-keys" Oct 31 03:34:44.551951 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.551864 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167d5711-a368-4260-a52d-07fc3f38b0e9" containerName="revoke-keys" Oct 31 03:34:44.552042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.551970 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="167d5711-a368-4260-a52d-07fc3f38b0e9" containerName="revoke-keys" Oct 31 03:34:44.556628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.556594 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.614391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.614354 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d"] Oct 31 03:34:44.661333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.661295 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:34:44.661333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.661306 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:34:44.661333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.661306 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:34:44.661578 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.661350 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:34:44.697611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.697567 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t85t\" (UniqueName: \"kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.697808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.697730 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.697808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.697779 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.799624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.799588 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.799830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.799653 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.799830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.799777 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t85t\" (UniqueName: \"kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.800455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.800424 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.802407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.802344 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.861737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.861695 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t85t\" (UniqueName: \"kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t\") pod \"maas-api-revoke-sub-e8ca67f3-dvd9d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:44.873582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:44.873555 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:45.112002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:45.111968 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj"] Oct 31 03:34:45.112236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:45.112211 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" containerID="cri-o://2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb" gracePeriod=30 Oct 31 03:34:45.254053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:45.254028 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d"] Oct 31 03:34:45.256573 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:34:45.256538 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670b7ee8_dedb_40f3_80f0_5c52ae241d8d.slice/crio-bdd3272fa4ff868bf156ce5fa70ff5b8782cc56417c84af9b9afb8fc023ed2d5 WatchSource:0}: Error finding container bdd3272fa4ff868bf156ce5fa70ff5b8782cc56417c84af9b9afb8fc023ed2d5: Status 404 returned error can't find the container with id bdd3272fa4ff868bf156ce5fa70ff5b8782cc56417c84af9b9afb8fc023ed2d5 Oct 31 03:34:45.654848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:45.654814 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" event={"ID":"670b7ee8-dedb-40f3-80f0-5c52ae241d8d","Type":"ContainerStarted","Data":"81fb2ad5b258932dee692864f996e45eafcb288c4ef8e0505db5e8306c62beea"} Oct 31 03:34:45.655357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:45.654861 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" event={"ID":"670b7ee8-dedb-40f3-80f0-5c52ae241d8d","Type":"ContainerStarted","Data":"bdd3272fa4ff868bf156ce5fa70ff5b8782cc56417c84af9b9afb8fc023ed2d5"} Oct 31 03:34:45.772732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:45.772608 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" podStartSLOduration=1.7725945840000001 podStartE2EDuration="1.772594584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:45.770679005 +0000 UTC m=+913.567397265" watchObservedRunningTime="2026-10-31 03:34:45.772594584 +0000 UTC m=+913.569312844" Oct 31 03:34:46.415331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:46.415293 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:46.662243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:46.662214 4488 generic.go:356] "Generic (PLEG): container finished" podID="670b7ee8-dedb-40f3-80f0-5c52ae241d8d" containerID="81fb2ad5b258932dee692864f996e45eafcb288c4ef8e0505db5e8306c62beea" exitCode=0 Oct 31 03:34:46.662642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:46.662307 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" event={"ID":"670b7ee8-dedb-40f3-80f0-5c52ae241d8d","Type":"ContainerDied","Data":"81fb2ad5b258932dee692864f996e45eafcb288c4ef8e0505db5e8306c62beea"} Oct 31 03:34:47.816657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.816621 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:47.938627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.938544 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca\") pod \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " Oct 31 03:34:47.938797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.938672 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca\") pod \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " Oct 31 03:34:47.938859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.938796 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t\" (UniqueName: \"kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t\") pod \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\" (UID: \"670b7ee8-dedb-40f3-80f0-5c52ae241d8d\") " Oct 31 03:34:47.938967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.938937 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca" pod "670b7ee8-dedb-40f3-80f0-5c52ae241d8d" (UID: "670b7ee8-dedb-40f3-80f0-5c52ae241d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:47.939469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.939342 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:47.941238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.941213 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca" pod "670b7ee8-dedb-40f3-80f0-5c52ae241d8d" (UID: "670b7ee8-dedb-40f3-80f0-5c52ae241d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:34:47.941327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:47.941287 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t" pod "670b7ee8-dedb-40f3-80f0-5c52ae241d8d" (UID: "670b7ee8-dedb-40f3-80f0-5c52ae241d8d"). InnerVolumeSpecName "kube-api-access-7t85t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:48.041155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:48.041070 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t85t\" (UniqueName: \"kubernetes.io/projected/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-kube-api-access-7t85t\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:48.041155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:48.041153 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670b7ee8-dedb-40f3-80f0-5c52ae241d8d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:48.676778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:48.676750 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" Oct 31 03:34:48.676951 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:48.676780 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d" event={"ID":"670b7ee8-dedb-40f3-80f0-5c52ae241d8d","Type":"ContainerDied","Data":"bdd3272fa4ff868bf156ce5fa70ff5b8782cc56417c84af9b9afb8fc023ed2d5"} Oct 31 03:34:48.676951 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:48.676829 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd3272fa4ff868bf156ce5fa70ff5b8782cc56417c84af9b9afb8fc023ed2d5" Oct 31 03:34:50.393724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.393698 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567034 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567149 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567208 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567232 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567264 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567295 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567357 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567450 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.567498 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket\") pod \"939b9f49-7813-4139-821b-950f2e92fc68\" (UID: \"939b9f49-7813-4139-821b-950f2e92fc68\") " Oct 31 03:34:50.568731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.568489 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:34:50.568731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.568576 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:50.568843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.568766 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:34:50.569062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.568940 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:34:50.569265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.569237 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:34:50.570942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.570914 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:34:50.571037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.570870 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "kube-api-access-bg9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:50.571037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.570958 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:50.571145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.571045 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo" pod "939b9f49-7813-4139-821b-950f2e92fc68" (UID: "939b9f49-7813-4139-821b-950f2e92fc68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:34:50.668996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.668887 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.668996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.668938 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.668996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.668954 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939b9f49-7813-4139-821b-950f2e92fc68-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.668996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.668968 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.668996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.668982 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.668996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.668996 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939b9f49-7813-4139-821b-950f2e92fc68-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.669493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.669010 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg9bw\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-kube-api-access-bg9bw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.669493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.669026 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939b9f49-7813-4139-821b-950f2e92fc68-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.669493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.669039 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939b9f49-7813-4139-821b-950f2e92fc68-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.689999 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.689962 4488 generic.go:356] "Generic (PLEG): container finished" podID="939b9f49-7813-4139-821b-950f2e92fc68" containerID="2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb" exitCode=0 Oct 31 03:34:50.690207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.690028 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" Oct 31 03:34:50.690207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.690048 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" event={"ID":"939b9f49-7813-4139-821b-950f2e92fc68","Type":"ContainerDied","Data":"2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb"} Oct 31 03:34:50.690207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.690162 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj" event={"ID":"939b9f49-7813-4139-821b-950f2e92fc68","Type":"ContainerDied","Data":"141b93b1f008335d9b4f3654ff3fa0a4256d14de81db7d22464ab77ecfdc15c2"} Oct 31 03:34:50.690207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.690188 4488 scope.go:122] "RemoveContainer" containerID="2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb" Oct 31 03:34:50.703537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.703518 4488 scope.go:122] "RemoveContainer" containerID="2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb" Oct 31 03:34:50.703840 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:34:50.703823 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb\": container with ID starting with 2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb not found: ID does not exist" containerID="2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb" Oct 31 03:34:50.703886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.703850 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb"} err="failed to get container status \"2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb\": rpc error: code = NotFound desc = could not find container \"2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb\": container with ID starting with 2d6db985e4a81c7a0058a5506517f38c21e235bb7db5e9f8c5fa55f4ce2e48bb not found: ID does not exist" Oct 31 03:34:50.856491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.856458 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj"] Oct 31 03:34:50.860718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:50.860686 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kbfzj"] Oct 31 03:34:51.350924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:51.350874 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:52.317718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:52.317687 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939b9f49-7813-4139-821b-950f2e92fc68" path="/var/lib/kubelet/pods/939b9f49-7813-4139-821b-950f2e92fc68/volumes" Oct 31 03:34:56.351005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:34:56.350960 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:01.355015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:01.354980 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:02.757869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:02.757830 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:35:06.317045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.317013 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:06.589874 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:35:06.589839 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:35:06.719887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.719856 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9"] Oct 31 03:35:06.720323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.720310 4488 cpu_manager.go:414] "containerMap: removing container" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" Oct 31 03:35:06.720375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.720325 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" Oct 31 03:35:06.720375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.720349 4488 cpu_manager.go:414] "containerMap: removing container" podUID="670b7ee8-dedb-40f3-80f0-5c52ae241d8d" containerName="revoke-keys" Oct 31 03:35:06.720375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.720354 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670b7ee8-dedb-40f3-80f0-5c52ae241d8d" containerName="revoke-keys" Oct 31 03:35:06.720465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.720423 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="939b9f49-7813-4139-821b-950f2e92fc68" containerName="istio-proxy" Oct 31 03:35:06.720465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.720436 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="670b7ee8-dedb-40f3-80f0-5c52ae241d8d" containerName="revoke-keys" Oct 31 03:35:06.725432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.725412 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.772992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.772944 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fr8\" (UniqueName: \"kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.772992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.772995 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.773279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.773080 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.778887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.778769 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:06.828423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.828386 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:06.828423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.828406 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:06.828655 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.828601 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:35:06.831014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.830994 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9"] Oct 31 03:35:06.873846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.873747 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fr8\" (UniqueName: \"kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.873846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.873794 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.873846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.873811 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.874555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.874521 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.876535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.876512 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:06.936264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:06.936229 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fr8\" (UniqueName: \"kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8\") pod \"maas-api-revoke-sub-e4977a29-lbdn9\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:07.039223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:07.039181 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:07.429078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:07.429055 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9"] Oct 31 03:35:07.431271 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:07.431238 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb088be40_10c1_43c9_a94e_e60a05190ee7.slice/crio-232762306e7a65c39ab6afd0ef57a125c3d4c802c8af557a03de53c4c18706c3 WatchSource:0}: Error finding container 232762306e7a65c39ab6afd0ef57a125c3d4c802c8af557a03de53c4c18706c3: Status 404 returned error can't find the container with id 232762306e7a65c39ab6afd0ef57a125c3d4c802c8af557a03de53c4c18706c3 Oct 31 03:35:07.798453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:07.798413 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" event={"ID":"b088be40-10c1-43c9-a94e-e60a05190ee7","Type":"ContainerStarted","Data":"618db120e6514d284694f56a7f2fd65dcb4dfe2a5f4699776761d14d3ab6718b"} Oct 31 03:35:07.798650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:07.798488 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" event={"ID":"b088be40-10c1-43c9-a94e-e60a05190ee7","Type":"ContainerStarted","Data":"232762306e7a65c39ab6afd0ef57a125c3d4c802c8af557a03de53c4c18706c3"} Oct 31 03:35:07.915311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:07.915261 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" podStartSLOduration=1.915246785 podStartE2EDuration="1.915246785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:07.914555175 +0000 UTC m=+935.711273434" watchObservedRunningTime="2026-10-31 03:35:07.915246785 +0000 UTC m=+935.711965097" Oct 31 03:35:08.652764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.652728 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx"] Oct 31 03:35:08.656428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.656411 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.715159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.715121 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx"] Oct 31 03:35:08.803051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.803012 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlncr\" (UniqueName: \"kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.803295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.803256 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.803378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.803337 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.805916 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.805890 4488 generic.go:356] "Generic (PLEG): container finished" podID="b088be40-10c1-43c9-a94e-e60a05190ee7" containerID="618db120e6514d284694f56a7f2fd65dcb4dfe2a5f4699776761d14d3ab6718b" exitCode=0 Oct 31 03:35:08.806041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.805985 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" event={"ID":"b088be40-10c1-43c9-a94e-e60a05190ee7","Type":"ContainerDied","Data":"618db120e6514d284694f56a7f2fd65dcb4dfe2a5f4699776761d14d3ab6718b"} Oct 31 03:35:08.904163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.904056 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.904163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.904126 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.904432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.904180 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlncr\" (UniqueName: \"kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.904778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.904752 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.906683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.906666 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.966006 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.965978 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlncr\" (UniqueName: \"kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr\") pod \"maas-api-revoke-sub-643713c8-jjknx\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:08.970853 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:08.970811 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:09.352293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:09.352264 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx"] Oct 31 03:35:09.354033 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:09.354008 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02603570_fcca_4fe6_9c0a_b67c402bdd1b.slice/crio-7294c19d178b5984e405418ceb7c1a3277f8ed6d513a875016fdc0bc93ee44bf WatchSource:0}: Error finding container 7294c19d178b5984e405418ceb7c1a3277f8ed6d513a875016fdc0bc93ee44bf: Status 404 returned error can't find the container with id 7294c19d178b5984e405418ceb7c1a3277f8ed6d513a875016fdc0bc93ee44bf Oct 31 03:35:09.813946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:09.813905 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" event={"ID":"02603570-fcca-4fe6-9c0a-b67c402bdd1b","Type":"ContainerStarted","Data":"8edcb717382271171ad8f1a34e09d8148d5991ef24f5cf541d9da49a06e00cfa"} Oct 31 03:35:09.814405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:09.813997 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" event={"ID":"02603570-fcca-4fe6-9c0a-b67c402bdd1b","Type":"ContainerStarted","Data":"7294c19d178b5984e405418ceb7c1a3277f8ed6d513a875016fdc0bc93ee44bf"} Oct 31 03:35:09.931687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:09.931642 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" podStartSLOduration=1.931626954 podStartE2EDuration="1.931626954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:09.929508594 +0000 UTC m=+937.726226853" watchObservedRunningTime="2026-10-31 03:35:09.931626954 +0000 UTC m=+937.728345211" Oct 31 03:35:09.948506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:09.948482 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:10.017477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.017441 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca\") pod \"b088be40-10c1-43c9-a94e-e60a05190ee7\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " Oct 31 03:35:10.017667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.017539 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8\" (UniqueName: \"kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8\") pod \"b088be40-10c1-43c9-a94e-e60a05190ee7\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " Oct 31 03:35:10.017667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.017598 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca\") pod \"b088be40-10c1-43c9-a94e-e60a05190ee7\" (UID: \"b088be40-10c1-43c9-a94e-e60a05190ee7\") " Oct 31 03:35:10.017977 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.017951 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca" pod "b088be40-10c1-43c9-a94e-e60a05190ee7" (UID: "b088be40-10c1-43c9-a94e-e60a05190ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:10.019660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.019631 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca" pod "b088be40-10c1-43c9-a94e-e60a05190ee7" (UID: "b088be40-10c1-43c9-a94e-e60a05190ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:10.019764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.019696 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8" pod "b088be40-10c1-43c9-a94e-e60a05190ee7" (UID: "b088be40-10c1-43c9-a94e-e60a05190ee7"). InnerVolumeSpecName "kube-api-access-v6fr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:10.118820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.118734 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.118820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.118766 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6fr8\" (UniqueName: \"kubernetes.io/projected/b088be40-10c1-43c9-a94e-e60a05190ee7-kube-api-access-v6fr8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.118820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.118779 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b088be40-10c1-43c9-a94e-e60a05190ee7-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:10.820230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.820197 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" Oct 31 03:35:10.820671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.820235 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9" event={"ID":"b088be40-10c1-43c9-a94e-e60a05190ee7","Type":"ContainerDied","Data":"232762306e7a65c39ab6afd0ef57a125c3d4c802c8af557a03de53c4c18706c3"} Oct 31 03:35:10.820671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.820291 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232762306e7a65c39ab6afd0ef57a125c3d4c802c8af557a03de53c4c18706c3" Oct 31 03:35:10.821901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.821871 4488 generic.go:356] "Generic (PLEG): container finished" podID="02603570-fcca-4fe6-9c0a-b67c402bdd1b" containerID="8edcb717382271171ad8f1a34e09d8148d5991ef24f5cf541d9da49a06e00cfa" exitCode=0 Oct 31 03:35:10.822024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:10.821963 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" event={"ID":"02603570-fcca-4fe6-9c0a-b67c402bdd1b","Type":"ContainerDied","Data":"8edcb717382271171ad8f1a34e09d8148d5991ef24f5cf541d9da49a06e00cfa"} Oct 31 03:35:11.123192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:11.123161 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:11.968602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:11.968567 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:12.037628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.037594 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr\" (UniqueName: \"kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr\") pod \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " Oct 31 03:35:12.037628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.037633 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca\") pod \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " Oct 31 03:35:12.037899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.037789 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca\") pod \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\" (UID: \"02603570-fcca-4fe6-9c0a-b67c402bdd1b\") " Oct 31 03:35:12.038179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.038140 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca" pod "02603570-fcca-4fe6-9c0a-b67c402bdd1b" (UID: "02603570-fcca-4fe6-9c0a-b67c402bdd1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:12.038450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.038230 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:12.039855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.039832 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr" pod "02603570-fcca-4fe6-9c0a-b67c402bdd1b" (UID: "02603570-fcca-4fe6-9c0a-b67c402bdd1b"). InnerVolumeSpecName "kube-api-access-mlncr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:12.039989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.039901 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca" pod "02603570-fcca-4fe6-9c0a-b67c402bdd1b" (UID: "02603570-fcca-4fe6-9c0a-b67c402bdd1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:12.139299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.139211 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlncr\" (UniqueName: \"kubernetes.io/projected/02603570-fcca-4fe6-9c0a-b67c402bdd1b-kube-api-access-mlncr\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:12.139299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.139240 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02603570-fcca-4fe6-9c0a-b67c402bdd1b-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:12.834526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.834496 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" event={"ID":"02603570-fcca-4fe6-9c0a-b67c402bdd1b","Type":"ContainerDied","Data":"7294c19d178b5984e405418ceb7c1a3277f8ed6d513a875016fdc0bc93ee44bf"} Oct 31 03:35:12.834715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.834530 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx" Oct 31 03:35:12.834715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:12.834544 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7294c19d178b5984e405418ceb7c1a3277f8ed6d513a875016fdc0bc93ee44bf" Oct 31 03:35:16.142218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.142130 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:16.423661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.423573 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6"] Oct 31 03:35:16.424036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.424021 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b088be40-10c1-43c9-a94e-e60a05190ee7" containerName="revoke-keys" Oct 31 03:35:16.424119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.424036 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b088be40-10c1-43c9-a94e-e60a05190ee7" containerName="revoke-keys" Oct 31 03:35:16.424119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.424046 4488 cpu_manager.go:414] "containerMap: removing container" podUID="02603570-fcca-4fe6-9c0a-b67c402bdd1b" containerName="revoke-keys" Oct 31 03:35:16.424119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.424051 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02603570-fcca-4fe6-9c0a-b67c402bdd1b" containerName="revoke-keys" Oct 31 03:35:16.424284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.424156 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b088be40-10c1-43c9-a94e-e60a05190ee7" containerName="revoke-keys" Oct 31 03:35:16.424284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.424166 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="02603570-fcca-4fe6-9c0a-b67c402bdd1b" containerName="revoke-keys" Oct 31 03:35:16.427483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.427463 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.485878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.485844 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6"] Oct 31 03:35:16.532425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.532395 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:16.532425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.532409 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:16.532425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.532413 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:16.532676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.532466 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:35:16.584867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.584837 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.584867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.584880 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.585144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.584899 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkxw\" (UniqueName: \"kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.686676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.686582 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.686676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.686629 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.686676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.686648 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkxw\" (UniqueName: \"kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.687262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.687237 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.689200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.689179 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:16.748607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:16.748572 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkxw\" (UniqueName: \"kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw\") pod \"maas-api-revoke-sub-5806d66f-nlhm6\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:17.040980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:17.040884 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:17.429321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:17.429290 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6"] Oct 31 03:35:17.430622 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:17.430596 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a55e462_2fcf_4c1c_b14e_395308653cb2.slice/crio-c088cadcab73d4e7c4a70f06242df2d17354e955af28cc52a5ec2f7858f17f0f WatchSource:0}: Error finding container c088cadcab73d4e7c4a70f06242df2d17354e955af28cc52a5ec2f7858f17f0f: Status 404 returned error can't find the container with id c088cadcab73d4e7c4a70f06242df2d17354e955af28cc52a5ec2f7858f17f0f Oct 31 03:35:17.862843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:17.862809 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" event={"ID":"9a55e462-2fcf-4c1c-b14e-395308653cb2","Type":"ContainerStarted","Data":"de97ef28f27127068d11a1a57732cdea9e50437e20f4fe98ed7db6f4fdfcd7c1"} Oct 31 03:35:17.863014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:17.862859 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" event={"ID":"9a55e462-2fcf-4c1c-b14e-395308653cb2","Type":"ContainerStarted","Data":"c088cadcab73d4e7c4a70f06242df2d17354e955af28cc52a5ec2f7858f17f0f"} Oct 31 03:35:17.981139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:17.980995 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" podStartSLOduration=1.9809786329999999 podStartE2EDuration="1.980978633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:17.978492245 +0000 UTC m=+945.775210504" watchObservedRunningTime="2026-10-31 03:35:17.980978633 +0000 UTC m=+945.777696892" Oct 31 03:35:18.869444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:18.869406 4488 generic.go:356] "Generic (PLEG): container finished" podID="9a55e462-2fcf-4c1c-b14e-395308653cb2" containerID="de97ef28f27127068d11a1a57732cdea9e50437e20f4fe98ed7db6f4fdfcd7c1" exitCode=0 Oct 31 03:35:18.869856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:18.869500 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" event={"ID":"9a55e462-2fcf-4c1c-b14e-395308653cb2","Type":"ContainerDied","Data":"de97ef28f27127068d11a1a57732cdea9e50437e20f4fe98ed7db6f4fdfcd7c1"} Oct 31 03:35:20.103607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.103571 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:20.227106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.226997 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca\") pod \"9a55e462-2fcf-4c1c-b14e-395308653cb2\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " Oct 31 03:35:20.228155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.228122 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw\" (UniqueName: \"kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw\") pod \"9a55e462-2fcf-4c1c-b14e-395308653cb2\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " Oct 31 03:35:20.228596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.228574 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca\") pod \"9a55e462-2fcf-4c1c-b14e-395308653cb2\" (UID: \"9a55e462-2fcf-4c1c-b14e-395308653cb2\") " Oct 31 03:35:20.229516 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.229167 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca" pod "9a55e462-2fcf-4c1c-b14e-395308653cb2" (UID: "9a55e462-2fcf-4c1c-b14e-395308653cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:20.234556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.234531 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca" pod "9a55e462-2fcf-4c1c-b14e-395308653cb2" (UID: "9a55e462-2fcf-4c1c-b14e-395308653cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:20.247076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.245670 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw" pod "9a55e462-2fcf-4c1c-b14e-395308653cb2" (UID: "9a55e462-2fcf-4c1c-b14e-395308653cb2"). InnerVolumeSpecName "kube-api-access-gbkxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:20.331045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.331005 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:20.331341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.331297 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkxw\" (UniqueName: \"kubernetes.io/projected/9a55e462-2fcf-4c1c-b14e-395308653cb2-kube-api-access-gbkxw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:20.331341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.331321 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a55e462-2fcf-4c1c-b14e-395308653cb2-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:20.881276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.881241 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" event={"ID":"9a55e462-2fcf-4c1c-b14e-395308653cb2","Type":"ContainerDied","Data":"c088cadcab73d4e7c4a70f06242df2d17354e955af28cc52a5ec2f7858f17f0f"} Oct 31 03:35:20.881276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.881260 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6" Oct 31 03:35:20.881483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:20.881294 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c088cadcab73d4e7c4a70f06242df2d17354e955af28cc52a5ec2f7858f17f0f" Oct 31 03:35:21.011881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:21.011847 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:21.554414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:21.554375 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:24.099713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.099676 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx"] Oct 31 03:35:24.100383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.100362 4488 cpu_manager.go:414] "containerMap: removing container" podUID="9a55e462-2fcf-4c1c-b14e-395308653cb2" containerName="revoke-keys" Oct 31 03:35:24.100467 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.100385 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a55e462-2fcf-4c1c-b14e-395308653cb2" containerName="revoke-keys" Oct 31 03:35:24.100575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.100562 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a55e462-2fcf-4c1c-b14e-395308653cb2" containerName="revoke-keys" Oct 31 03:35:24.103961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.103920 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.209033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.209002 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:24.209033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.209032 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:24.209278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.209032 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:24.209278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.209163 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:35:24.211966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.211923 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx"] Oct 31 03:35:24.288263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.288220 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.288494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.288365 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2csd\" (UniqueName: \"kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.288494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.288425 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.389919 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.389825 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2csd\" (UniqueName: \"kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.389919 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.389878 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.390135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.389949 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.390610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.390585 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.392852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.392829 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.451107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.451053 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2csd\" (UniqueName: \"kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd\") pod \"maas-api-revoke-sub-6c10b9bf-8gljx\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:24.718610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:24.718509 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:25.101247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:25.101222 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx"] Oct 31 03:35:25.102784 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:25.102761 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e550453_c9ea_4bb7_af38_a41f24fa6d0f.slice/crio-79196deebe666e7ba4438f5a62a9c1dffd8f820a99382b30e9788d66d8d09ca5 WatchSource:0}: Error finding container 79196deebe666e7ba4438f5a62a9c1dffd8f820a99382b30e9788d66d8d09ca5: Status 404 returned error can't find the container with id 79196deebe666e7ba4438f5a62a9c1dffd8f820a99382b30e9788d66d8d09ca5 Oct 31 03:35:25.821332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:25.821250 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:25.911643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:25.911606 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" event={"ID":"8e550453-c9ea-4bb7-af38-a41f24fa6d0f","Type":"ContainerStarted","Data":"7d11f9034af08adec2c3dbd621099da37a07e7563b7480ea4c2f8764d0650223"} Oct 31 03:35:25.911789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:25.911670 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" event={"ID":"8e550453-c9ea-4bb7-af38-a41f24fa6d0f","Type":"ContainerStarted","Data":"79196deebe666e7ba4438f5a62a9c1dffd8f820a99382b30e9788d66d8d09ca5"} Oct 31 03:35:26.030061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.029974 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" podStartSLOduration=2.029952744 podStartE2EDuration="2.029952744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:26.027202543 +0000 UTC m=+953.823920801" watchObservedRunningTime="2026-10-31 03:35:26.029952744 +0000 UTC m=+953.826671003" Oct 31 03:35:26.412107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.412061 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9"] Oct 31 03:35:26.415850 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.415827 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.473786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.473751 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9"] Oct 31 03:35:26.510751 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.510708 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.510974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.510834 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.510974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.510869 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7wm\" (UniqueName: \"kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.523052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.523013 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n"] Oct 31 03:35:26.526924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.526906 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.571220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.571182 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n"] Oct 31 03:35:26.615190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.612666 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.615190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.612809 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.615190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.612848 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7wm\" (UniqueName: \"kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.618401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.618361 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.633681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.633184 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.674509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.674479 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7wm\" (UniqueName: \"kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm\") pod \"maas-api-revoke-sub-ef41777b-8c4k9\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.714135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.714079 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.714135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.714142 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmcg\" (UniqueName: \"kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.714355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.714301 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.730157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.730119 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:26.815624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.815586 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.815776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.815633 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmcg\" (UniqueName: \"kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.815776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.815683 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.816530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.816502 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.818425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.818352 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.877620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.877542 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmcg\" (UniqueName: \"kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg\") pod \"maas-api-revoke-sub-a977a059-zzd6n\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:26.923414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.923376 4488 generic.go:356] "Generic (PLEG): container finished" podID="8e550453-c9ea-4bb7-af38-a41f24fa6d0f" containerID="7d11f9034af08adec2c3dbd621099da37a07e7563b7480ea4c2f8764d0650223" exitCode=0 Oct 31 03:35:26.923601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:26.923455 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" event={"ID":"8e550453-c9ea-4bb7-af38-a41f24fa6d0f","Type":"ContainerDied","Data":"7d11f9034af08adec2c3dbd621099da37a07e7563b7480ea4c2f8764d0650223"} Oct 31 03:35:27.113841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:27.113687 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9"] Oct 31 03:35:27.117243 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:27.117210 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782730d8_42b1_41db_9fd1_4086d6a95026.slice/crio-56c084491c08f9da38f8aeed69e0d3e0b26a8e94e2a3f6b1c116375074bcc3c6 WatchSource:0}: Error finding container 56c084491c08f9da38f8aeed69e0d3e0b26a8e94e2a3f6b1c116375074bcc3c6: Status 404 returned error can't find the container with id 56c084491c08f9da38f8aeed69e0d3e0b26a8e94e2a3f6b1c116375074bcc3c6 Oct 31 03:35:27.142306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:27.142277 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:27.552673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:27.552632 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n"] Oct 31 03:35:27.559268 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:27.559224 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96441321_406e_45cd_8238_d690de5e21fb.slice/crio-36852c0be57108bb7e321f73689487ad7ff7ac7e4f4e17f297dae78ac88d7290 WatchSource:0}: Error finding container 36852c0be57108bb7e321f73689487ad7ff7ac7e4f4e17f297dae78ac88d7290: Status 404 returned error can't find the container with id 36852c0be57108bb7e321f73689487ad7ff7ac7e4f4e17f297dae78ac88d7290 Oct 31 03:35:27.941112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:27.938260 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" event={"ID":"782730d8-42b1-41db-9fd1-4086d6a95026","Type":"ContainerStarted","Data":"5d8bd624467e0d4707dbb650c4b8250a1da0369f99a5bdbb3168ac60e0cd8a5e"} Oct 31 03:35:27.941112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:27.938331 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" event={"ID":"782730d8-42b1-41db-9fd1-4086d6a95026","Type":"ContainerStarted","Data":"56c084491c08f9da38f8aeed69e0d3e0b26a8e94e2a3f6b1c116375074bcc3c6"} Oct 31 03:35:27.941391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:27.941368 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" event={"ID":"96441321-406e-45cd-8238-d690de5e21fb","Type":"ContainerStarted","Data":"36852c0be57108bb7e321f73689487ad7ff7ac7e4f4e17f297dae78ac88d7290"} Oct 31 03:35:28.062072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.061996 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" podStartSLOduration=2.0619733670000002 podStartE2EDuration="2.061973367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:28.061438225 +0000 UTC m=+955.858156484" watchObservedRunningTime="2026-10-31 03:35:28.061973367 +0000 UTC m=+955.858691625" Oct 31 03:35:28.230880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.230846 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:28.248314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.248269 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca\") pod \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " Oct 31 03:35:28.248495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.248361 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd\" (UniqueName: \"kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd\") pod \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " Oct 31 03:35:28.248592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.248530 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca\") pod \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\" (UID: \"8e550453-c9ea-4bb7-af38-a41f24fa6d0f\") " Oct 31 03:35:28.248684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.248654 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca" pod "8e550453-c9ea-4bb7-af38-a41f24fa6d0f" (UID: "8e550453-c9ea-4bb7-af38-a41f24fa6d0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:28.249007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.248975 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:28.250898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.250868 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca" pod "8e550453-c9ea-4bb7-af38-a41f24fa6d0f" (UID: "8e550453-c9ea-4bb7-af38-a41f24fa6d0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:28.251442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.251349 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd" pod "8e550453-c9ea-4bb7-af38-a41f24fa6d0f" (UID: "8e550453-c9ea-4bb7-af38-a41f24fa6d0f"). InnerVolumeSpecName "kube-api-access-k2csd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:28.350788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.350597 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2csd\" (UniqueName: \"kubernetes.io/projected/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-kube-api-access-k2csd\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:28.350788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.350626 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e550453-c9ea-4bb7-af38-a41f24fa6d0f-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:28.948720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.948630 4488 generic.go:356] "Generic (PLEG): container finished" podID="96441321-406e-45cd-8238-d690de5e21fb" containerID="54a9e842b51a08bb86b7bb4e6b17b5299982130ac7d02c6f61b55e072c112ffb" exitCode=0 Oct 31 03:35:28.949155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.948710 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" event={"ID":"96441321-406e-45cd-8238-d690de5e21fb","Type":"ContainerDied","Data":"54a9e842b51a08bb86b7bb4e6b17b5299982130ac7d02c6f61b55e072c112ffb"} Oct 31 03:35:28.950194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.950174 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" Oct 31 03:35:28.950313 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.950192 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx" event={"ID":"8e550453-c9ea-4bb7-af38-a41f24fa6d0f","Type":"ContainerDied","Data":"79196deebe666e7ba4438f5a62a9c1dffd8f820a99382b30e9788d66d8d09ca5"} Oct 31 03:35:28.950313 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.950235 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79196deebe666e7ba4438f5a62a9c1dffd8f820a99382b30e9788d66d8d09ca5" Oct 31 03:35:28.951826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.951802 4488 generic.go:356] "Generic (PLEG): container finished" podID="782730d8-42b1-41db-9fd1-4086d6a95026" containerID="5d8bd624467e0d4707dbb650c4b8250a1da0369f99a5bdbb3168ac60e0cd8a5e" exitCode=0 Oct 31 03:35:28.951959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:28.951836 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" event={"ID":"782730d8-42b1-41db-9fd1-4086d6a95026","Type":"ContainerDied","Data":"5d8bd624467e0d4707dbb650c4b8250a1da0369f99a5bdbb3168ac60e0cd8a5e"} Oct 31 03:35:30.121290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.121268 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:30.124522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.124503 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:30.166503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166472 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg\" (UniqueName: \"kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg\") pod \"96441321-406e-45cd-8238-d690de5e21fb\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " Oct 31 03:35:30.166662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166517 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca\") pod \"96441321-406e-45cd-8238-d690de5e21fb\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " Oct 31 03:35:30.166662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166554 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm\" (UniqueName: \"kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm\") pod \"782730d8-42b1-41db-9fd1-4086d6a95026\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " Oct 31 03:35:30.166662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166603 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca\") pod \"782730d8-42b1-41db-9fd1-4086d6a95026\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " Oct 31 03:35:30.166662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166632 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca\") pod \"96441321-406e-45cd-8238-d690de5e21fb\" (UID: \"96441321-406e-45cd-8238-d690de5e21fb\") " Oct 31 03:35:30.166952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166684 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca\") pod \"782730d8-42b1-41db-9fd1-4086d6a95026\" (UID: \"782730d8-42b1-41db-9fd1-4086d6a95026\") " Oct 31 03:35:30.167013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.166963 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca" pod "96441321-406e-45cd-8238-d690de5e21fb" (UID: "96441321-406e-45cd-8238-d690de5e21fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:30.167228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.167207 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96441321-406e-45cd-8238-d690de5e21fb-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:30.167478 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.167442 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca" pod "782730d8-42b1-41db-9fd1-4086d6a95026" (UID: "782730d8-42b1-41db-9fd1-4086d6a95026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:30.168889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.168859 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm" pod "782730d8-42b1-41db-9fd1-4086d6a95026" (UID: "782730d8-42b1-41db-9fd1-4086d6a95026"). InnerVolumeSpecName "kube-api-access-9g7wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:30.168994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.168910 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg" pod "96441321-406e-45cd-8238-d690de5e21fb" (UID: "96441321-406e-45cd-8238-d690de5e21fb"). InnerVolumeSpecName "kube-api-access-klmcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:30.169372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.169355 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca" pod "782730d8-42b1-41db-9fd1-4086d6a95026" (UID: "782730d8-42b1-41db-9fd1-4086d6a95026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:30.169449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.169356 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca" pod "96441321-406e-45cd-8238-d690de5e21fb" (UID: "96441321-406e-45cd-8238-d690de5e21fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:30.267867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.267767 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmcg\" (UniqueName: \"kubernetes.io/projected/96441321-406e-45cd-8238-d690de5e21fb-kube-api-access-klmcg\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:30.267867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.267798 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7wm\" (UniqueName: \"kubernetes.io/projected/782730d8-42b1-41db-9fd1-4086d6a95026-kube-api-access-9g7wm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:30.267867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.267809 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:30.267867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.267819 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96441321-406e-45cd-8238-d690de5e21fb-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:30.267867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.267829 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782730d8-42b1-41db-9fd1-4086d6a95026-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:30.705798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.705758 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:30.964755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.964667 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" Oct 31 03:35:30.964755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.964667 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9" event={"ID":"782730d8-42b1-41db-9fd1-4086d6a95026","Type":"ContainerDied","Data":"56c084491c08f9da38f8aeed69e0d3e0b26a8e94e2a3f6b1c116375074bcc3c6"} Oct 31 03:35:30.964957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.964801 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c084491c08f9da38f8aeed69e0d3e0b26a8e94e2a3f6b1c116375074bcc3c6" Oct 31 03:35:30.966188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.966162 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" Oct 31 03:35:30.966305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.966209 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n" event={"ID":"96441321-406e-45cd-8238-d690de5e21fb","Type":"ContainerDied","Data":"36852c0be57108bb7e321f73689487ad7ff7ac7e4f4e17f297dae78ac88d7290"} Oct 31 03:35:30.966305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:30.966255 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36852c0be57108bb7e321f73689487ad7ff7ac7e4f4e17f297dae78ac88d7290" Oct 31 03:35:32.403197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403157 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7"] Oct 31 03:35:32.403564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403551 4488 cpu_manager.go:414] "containerMap: removing container" podUID="782730d8-42b1-41db-9fd1-4086d6a95026" containerName="revoke-keys" Oct 31 03:35:32.403564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403562 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782730d8-42b1-41db-9fd1-4086d6a95026" containerName="revoke-keys" Oct 31 03:35:32.403659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403580 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8e550453-c9ea-4bb7-af38-a41f24fa6d0f" containerName="revoke-keys" Oct 31 03:35:32.403659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403585 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e550453-c9ea-4bb7-af38-a41f24fa6d0f" containerName="revoke-keys" Oct 31 03:35:32.403659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403606 4488 cpu_manager.go:414] "containerMap: removing container" podUID="96441321-406e-45cd-8238-d690de5e21fb" containerName="revoke-keys" Oct 31 03:35:32.403659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403625 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96441321-406e-45cd-8238-d690de5e21fb" containerName="revoke-keys" Oct 31 03:35:32.403773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403685 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="96441321-406e-45cd-8238-d690de5e21fb" containerName="revoke-keys" Oct 31 03:35:32.403773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403696 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="782730d8-42b1-41db-9fd1-4086d6a95026" containerName="revoke-keys" Oct 31 03:35:32.403773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.403703 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e550453-c9ea-4bb7-af38-a41f24fa6d0f" containerName="revoke-keys" Oct 31 03:35:32.407211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.407189 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.462005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.461951 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:32.462005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.462017 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:32.471638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.471614 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:32.471638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.471651 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:35:32.489393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.489358 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.489572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.489397 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kd9\" (UniqueName: \"kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.489572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.489553 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.514760 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.514725 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7"] Oct 31 03:35:32.590566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.590529 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.590566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.590570 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kd9\" (UniqueName: \"kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.590818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.590621 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.591442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.591418 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.593248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.593230 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.652703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.652667 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kd9\" (UniqueName: \"kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9\") pod \"maas-api-revoke-sub-ffab464f-ppjp7\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:32.722940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:32.722849 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:33.099678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:33.099646 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7"] Oct 31 03:35:33.101568 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:33.101538 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374c30dc_c7ca_4ec7_b2c6_7fbb3dd94312.slice/crio-83d798ffc51fa8a72d67c234f656c295a91c1ac43ead6ea8643c1952f3ea58a2 WatchSource:0}: Error finding container 83d798ffc51fa8a72d67c234f656c295a91c1ac43ead6ea8643c1952f3ea58a2: Status 404 returned error can't find the container with id 83d798ffc51fa8a72d67c234f656c295a91c1ac43ead6ea8643c1952f3ea58a2 Oct 31 03:35:33.984562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:33.984523 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" event={"ID":"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312","Type":"ContainerStarted","Data":"c3d302c5ac2ba70a1039f018e830c62964f602065c18b08c25f8ca6ed4c5b5e6"} Oct 31 03:35:33.984975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:33.984575 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" event={"ID":"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312","Type":"ContainerStarted","Data":"83d798ffc51fa8a72d67c234f656c295a91c1ac43ead6ea8643c1952f3ea58a2"} Oct 31 03:35:34.102083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.102020 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" podStartSLOduration=2.101998327 podStartE2EDuration="2.101998327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:34.100112161 +0000 UTC m=+961.896830412" watchObservedRunningTime="2026-10-31 03:35:34.101998327 +0000 UTC m=+961.898716589" Oct 31 03:35:34.888905 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.888868 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4"] Oct 31 03:35:34.892620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.892597 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:34.919190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.919158 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:34.919190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.919207 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:34.919422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.919314 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzvs\" (UniqueName: \"kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:34.951359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.951322 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4"] Oct 31 03:35:34.993349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.993254 4488 generic.go:356] "Generic (PLEG): container finished" podID="374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" containerID="c3d302c5ac2ba70a1039f018e830c62964f602065c18b08c25f8ca6ed4c5b5e6" exitCode=0 Oct 31 03:35:34.993349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:34.993333 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" event={"ID":"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312","Type":"ContainerDied","Data":"c3d302c5ac2ba70a1039f018e830c62964f602065c18b08c25f8ca6ed4c5b5e6"} Oct 31 03:35:35.020278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.020237 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.020562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.020538 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.020728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.020712 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzvs\" (UniqueName: \"kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.022004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.021961 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.024480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.024458 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.082775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.082738 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzvs\" (UniqueName: \"kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs\") pod \"maas-api-revoke-sub-a2ef050d-cbnz4\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.206816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.206724 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:35.585216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.585188 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4"] Oct 31 03:35:35.586759 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:35.586728 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4533698d_250f_4cb7_9b7c_711174882642.slice/crio-4a0f18d734e61cba2c40b09724a79fb5be2f0485acc06849e1d4f26b904b9a8e WatchSource:0}: Error finding container 4a0f18d734e61cba2c40b09724a79fb5be2f0485acc06849e1d4f26b904b9a8e: Status 404 returned error can't find the container with id 4a0f18d734e61cba2c40b09724a79fb5be2f0485acc06849e1d4f26b904b9a8e Oct 31 03:35:35.710133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:35.710069 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:36.001675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.001641 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" event={"ID":"4533698d-250f-4cb7-9b7c-711174882642","Type":"ContainerStarted","Data":"5df21f92a67510e9f87ccf83a8163c37e5ad847efc6b1e75dc0f8dc45e6b374a"} Oct 31 03:35:36.002180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.001711 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" event={"ID":"4533698d-250f-4cb7-9b7c-711174882642","Type":"ContainerStarted","Data":"4a0f18d734e61cba2c40b09724a79fb5be2f0485acc06849e1d4f26b904b9a8e"} Oct 31 03:35:36.121061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.121003 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" podStartSLOduration=2.120984479 podStartE2EDuration="2.120984479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:36.118456107 +0000 UTC m=+963.915174367" watchObservedRunningTime="2026-10-31 03:35:36.120984479 +0000 UTC m=+963.917702773" Oct 31 03:35:36.146552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.146516 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:36.234480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.234386 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca\") pod \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " Oct 31 03:35:36.234645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.234604 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9\" (UniqueName: \"kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9\") pod \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " Oct 31 03:35:36.234645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.234635 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca\") pod \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\" (UID: \"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312\") " Oct 31 03:35:36.234790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.234764 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca" pod "374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" (UID: "374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:36.235045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.235028 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:36.237034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.237009 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca" pod "374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" (UID: "374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:36.237165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.237071 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9" pod "374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" (UID: "374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312"). InnerVolumeSpecName "kube-api-access-v2kd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:36.336328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.336284 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2kd9\" (UniqueName: \"kubernetes.io/projected/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-kube-api-access-v2kd9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:36.336328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:36.336324 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:36.589842 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:35:36.589806 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:35:37.010170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:37.010138 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" Oct 31 03:35:37.010170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:37.010156 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7" event={"ID":"374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312","Type":"ContainerDied","Data":"83d798ffc51fa8a72d67c234f656c295a91c1ac43ead6ea8643c1952f3ea58a2"} Oct 31 03:35:37.010721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:37.010211 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d798ffc51fa8a72d67c234f656c295a91c1ac43ead6ea8643c1952f3ea58a2" Oct 31 03:35:37.013127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:37.012163 4488 generic.go:356] "Generic (PLEG): container finished" podID="4533698d-250f-4cb7-9b7c-711174882642" containerID="5df21f92a67510e9f87ccf83a8163c37e5ad847efc6b1e75dc0f8dc45e6b374a" exitCode=0 Oct 31 03:35:37.013127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:37.012405 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" event={"ID":"4533698d-250f-4cb7-9b7c-711174882642","Type":"ContainerDied","Data":"5df21f92a67510e9f87ccf83a8163c37e5ad847efc6b1e75dc0f8dc45e6b374a"} Oct 31 03:35:38.178922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.178899 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:38.255045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.255009 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs\" (UniqueName: \"kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs\") pod \"4533698d-250f-4cb7-9b7c-711174882642\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " Oct 31 03:35:38.255229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.255062 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca\") pod \"4533698d-250f-4cb7-9b7c-711174882642\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " Oct 31 03:35:38.255229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.255185 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca\") pod \"4533698d-250f-4cb7-9b7c-711174882642\" (UID: \"4533698d-250f-4cb7-9b7c-711174882642\") " Oct 31 03:35:38.255551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.255522 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca" pod "4533698d-250f-4cb7-9b7c-711174882642" (UID: "4533698d-250f-4cb7-9b7c-711174882642"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:38.255633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.255617 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4533698d-250f-4cb7-9b7c-711174882642-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:38.257568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.257538 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca" pod "4533698d-250f-4cb7-9b7c-711174882642" (UID: "4533698d-250f-4cb7-9b7c-711174882642"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:38.257568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.257555 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs" pod "4533698d-250f-4cb7-9b7c-711174882642" (UID: "4533698d-250f-4cb7-9b7c-711174882642"). InnerVolumeSpecName "kube-api-access-ztzvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:38.356588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.356547 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzvs\" (UniqueName: \"kubernetes.io/projected/4533698d-250f-4cb7-9b7c-711174882642-kube-api-access-ztzvs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:38.356588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:38.356588 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4533698d-250f-4cb7-9b7c-711174882642-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:39.024717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.024686 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" Oct 31 03:35:39.024717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.024708 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4" event={"ID":"4533698d-250f-4cb7-9b7c-711174882642","Type":"ContainerDied","Data":"4a0f18d734e61cba2c40b09724a79fb5be2f0485acc06849e1d4f26b904b9a8e"} Oct 31 03:35:39.024943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.024751 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a0f18d734e61cba2c40b09724a79fb5be2f0485acc06849e1d4f26b904b9a8e" Oct 31 03:35:39.307342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.307253 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z"] Oct 31 03:35:39.307924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.307891 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4533698d-250f-4cb7-9b7c-711174882642" containerName="revoke-keys" Oct 31 03:35:39.308010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.307928 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4533698d-250f-4cb7-9b7c-711174882642" containerName="revoke-keys" Oct 31 03:35:39.308010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.307989 4488 cpu_manager.go:414] "containerMap: removing container" podUID="374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" containerName="revoke-keys" Oct 31 03:35:39.308010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.307997 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" containerName="revoke-keys" Oct 31 03:35:39.308186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.308142 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4533698d-250f-4cb7-9b7c-711174882642" containerName="revoke-keys" Oct 31 03:35:39.308186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.308165 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" containerName="revoke-keys" Oct 31 03:35:39.312388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.312347 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.366131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.366038 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"default-dockercfg-f7wzm\"" Oct 31 03:35:39.366329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.366212 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"distinct-w2-dec4a0-kserve-self-signed-certs\"" Oct 31 03:35:39.366805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.366776 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphx2\" (UniqueName: \"kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.366921 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.366895 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.367036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.367014 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.367311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.367293 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.367469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.367446 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.367592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.367551 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.415598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.415557 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:39.415792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.415771 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"openshift-service-ca.crt\"" Oct 31 03:35:39.415839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.415803 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"kube-root-ca.crt\"" Oct 31 03:35:39.418803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.418782 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z"] Oct 31 03:35:39.468539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.468509 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.468539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.468546 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.468786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.468591 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.468786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.468627 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphx2\" (UniqueName: \"kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.468786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.468651 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.468786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.468704 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.469025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.469000 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.469135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.469052 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.469253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.469232 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.471964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.471887 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.480559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.480536 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.529957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.529926 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphx2\" (UniqueName: \"kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2\") pod \"distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:39.636115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:39.632022 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:40.038651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:40.038627 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z"] Oct 31 03:35:40.039697 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:40.039664 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc61b3003_fce7_4b05_bc06_71d024715ad7.slice/crio-476f3bef09e75952d58ebceca7ed5b8ab5febd0ca1a4616a92f4f0c222b8e94e WatchSource:0}: Error finding container 476f3bef09e75952d58ebceca7ed5b8ab5febd0ca1a4616a92f4f0c222b8e94e: Status 404 returned error can't find the container with id 476f3bef09e75952d58ebceca7ed5b8ab5febd0ca1a4616a92f4f0c222b8e94e Oct 31 03:35:40.542239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:40.542206 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:41.039460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:41.039420 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" event={"ID":"c61b3003-fce7-4b05-bc06-71d024715ad7","Type":"ContainerStarted","Data":"d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a"} Oct 31 03:35:41.039662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:41.039473 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" event={"ID":"c61b3003-fce7-4b05-bc06-71d024715ad7","Type":"ContainerStarted","Data":"476f3bef09e75952d58ebceca7ed5b8ab5febd0ca1a4616a92f4f0c222b8e94e"} Oct 31 03:35:41.158948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:41.158896 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podStartSLOduration=2.158881922 podStartE2EDuration="2.158881922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:41.156850645 +0000 UTC m=+968.953568904" watchObservedRunningTime="2026-10-31 03:35:41.158881922 +0000 UTC m=+968.955600221" Oct 31 03:35:41.637139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:41.637068 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:42.177692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.177650 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7"] Oct 31 03:35:42.181384 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.181362 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.236219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.236180 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:42.236802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.236778 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:42.285942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.285897 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:42.285942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.285936 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:35:42.288792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.288765 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7"] Oct 31 03:35:42.300148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.300117 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.300261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.300233 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.300384 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.300370 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgx4\" (UniqueName: \"kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.401436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.401400 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgx4\" (UniqueName: \"kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.401629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.401485 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.401629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.401524 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.402208 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.402181 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.404026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.404001 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.463038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.462967 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgx4\" (UniqueName: \"kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4\") pod \"maas-api-revoke-sub-d55548e9-8p4d7\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.494927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.494891 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:42.698183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.698149 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw"] Oct 31 03:35:42.702984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.702968 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.762645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.762569 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw"] Oct 31 03:35:42.805565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.805526 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfc2\" (UniqueName: \"kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.805752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.805676 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.805752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.805736 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.874950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.874917 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7"] Oct 31 03:35:42.876184 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:42.876156 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73b3f32_3340_4e62_9bac_20284e921d61.slice/crio-f8b830409740a59fe59fea2070659f5ba890f47d63ef2a98c894c13fcdb7ea0f WatchSource:0}: Error finding container f8b830409740a59fe59fea2070659f5ba890f47d63ef2a98c894c13fcdb7ea0f: Status 404 returned error can't find the container with id f8b830409740a59fe59fea2070659f5ba890f47d63ef2a98c894c13fcdb7ea0f Oct 31 03:35:42.907332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.907300 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.907465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.907430 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfc2\" (UniqueName: \"kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.907541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.907525 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.907972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.907952 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.909990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.909972 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:42.971222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:42.971182 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfc2\" (UniqueName: \"kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2\") pod \"maas-api-revoke-sub-37d7a03d-qjxhw\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:43.017452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:43.017366 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:43.052761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:43.052722 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" event={"ID":"b73b3f32-3340-4e62-9bac-20284e921d61","Type":"ContainerStarted","Data":"f8b830409740a59fe59fea2070659f5ba890f47d63ef2a98c894c13fcdb7ea0f"} Oct 31 03:35:43.395855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:43.395832 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw"] Oct 31 03:35:43.398228 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:43.398198 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00f1c3e_401f_461c_811f_71a6cf5ba1eb.slice/crio-a4fba457673eeb65f16325e5459dfc05e78e90875457c0a0a68303fbdcc4f51e WatchSource:0}: Error finding container a4fba457673eeb65f16325e5459dfc05e78e90875457c0a0a68303fbdcc4f51e: Status 404 returned error can't find the container with id a4fba457673eeb65f16325e5459dfc05e78e90875457c0a0a68303fbdcc4f51e Oct 31 03:35:44.060246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:44.060145 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" event={"ID":"c00f1c3e-401f-461c-811f-71a6cf5ba1eb","Type":"ContainerStarted","Data":"3ba10ae607e9d8d6a20117bf86d8f5c6d8529625299e3a469a9aa6e4ba54a534"} Oct 31 03:35:44.060246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:44.060205 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" event={"ID":"c00f1c3e-401f-461c-811f-71a6cf5ba1eb","Type":"ContainerStarted","Data":"a4fba457673eeb65f16325e5459dfc05e78e90875457c0a0a68303fbdcc4f51e"} Oct 31 03:35:44.061761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:44.061735 4488 generic.go:356] "Generic (PLEG): container finished" podID="b73b3f32-3340-4e62-9bac-20284e921d61" containerID="826a4005f84b16506a95334ae68e7d10633bcc39195bc1a14fc87007d88acf75" exitCode=0 Oct 31 03:35:44.061826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:44.061794 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" event={"ID":"b73b3f32-3340-4e62-9bac-20284e921d61","Type":"ContainerDied","Data":"826a4005f84b16506a95334ae68e7d10633bcc39195bc1a14fc87007d88acf75"} Oct 31 03:35:44.179342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:44.179286 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" podStartSLOduration=2.179269997 podStartE2EDuration="2.179269997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:44.176148244 +0000 UTC m=+971.972866507" watchObservedRunningTime="2026-10-31 03:35:44.179269997 +0000 UTC m=+971.975988254" Oct 31 03:35:45.035574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.035538 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.069560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.069527 4488 generic.go:356] "Generic (PLEG): container finished" podID="c00f1c3e-401f-461c-811f-71a6cf5ba1eb" containerID="3ba10ae607e9d8d6a20117bf86d8f5c6d8529625299e3a469a9aa6e4ba54a534" exitCode=0 Oct 31 03:35:45.069978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.069614 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" event={"ID":"c00f1c3e-401f-461c-811f-71a6cf5ba1eb","Type":"ContainerDied","Data":"3ba10ae607e9d8d6a20117bf86d8f5c6d8529625299e3a469a9aa6e4ba54a534"} Oct 31 03:35:45.193074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.193042 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.218464 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.218435 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:45.338199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.336234 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca\") pod \"b73b3f32-3340-4e62-9bac-20284e921d61\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " Oct 31 03:35:45.338199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.336291 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4\" (UniqueName: \"kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4\") pod \"b73b3f32-3340-4e62-9bac-20284e921d61\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " Oct 31 03:35:45.338199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.336439 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca\") pod \"b73b3f32-3340-4e62-9bac-20284e921d61\" (UID: \"b73b3f32-3340-4e62-9bac-20284e921d61\") " Oct 31 03:35:45.338199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.336953 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca" pod "b73b3f32-3340-4e62-9bac-20284e921d61" (UID: "b73b3f32-3340-4e62-9bac-20284e921d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:45.343215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.343183 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4" pod "b73b3f32-3340-4e62-9bac-20284e921d61" (UID: "b73b3f32-3340-4e62-9bac-20284e921d61"). InnerVolumeSpecName "kube-api-access-djgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:45.343215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.343198 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca" pod "b73b3f32-3340-4e62-9bac-20284e921d61" (UID: "b73b3f32-3340-4e62-9bac-20284e921d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:45.408721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.408684 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.437897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.437853 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:45.437897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.437888 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgx4\" (UniqueName: \"kubernetes.io/projected/b73b3f32-3340-4e62-9bac-20284e921d61-kube-api-access-djgx4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:45.437897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:45.437898 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b3f32-3340-4e62-9bac-20284e921d61-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:46.076323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.076294 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" Oct 31 03:35:46.076830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.076331 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7" event={"ID":"b73b3f32-3340-4e62-9bac-20284e921d61","Type":"ContainerDied","Data":"f8b830409740a59fe59fea2070659f5ba890f47d63ef2a98c894c13fcdb7ea0f"} Oct 31 03:35:46.076830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.076388 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b830409740a59fe59fea2070659f5ba890f47d63ef2a98c894c13fcdb7ea0f" Oct 31 03:35:46.224811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.224782 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:46.347054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.347019 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca\") pod \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " Oct 31 03:35:46.347247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.347068 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca\") pod \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " Oct 31 03:35:46.347247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.347204 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2\" (UniqueName: \"kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2\") pod \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\" (UID: \"c00f1c3e-401f-461c-811f-71a6cf5ba1eb\") " Oct 31 03:35:46.347513 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.347481 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca" pod "c00f1c3e-401f-461c-811f-71a6cf5ba1eb" (UID: "c00f1c3e-401f-461c-811f-71a6cf5ba1eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:46.347701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.347678 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:46.351247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.351213 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca" pod "c00f1c3e-401f-461c-811f-71a6cf5ba1eb" (UID: "c00f1c3e-401f-461c-811f-71a6cf5ba1eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:46.356281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.356249 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2" pod "c00f1c3e-401f-461c-811f-71a6cf5ba1eb" (UID: "c00f1c3e-401f-461c-811f-71a6cf5ba1eb"). InnerVolumeSpecName "kube-api-access-wmfc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:46.448833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.448794 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:46.448833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:46.448829 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmfc2\" (UniqueName: \"kubernetes.io/projected/c00f1c3e-401f-461c-811f-71a6cf5ba1eb-kube-api-access-wmfc2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:47.083384 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:47.083353 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" Oct 31 03:35:47.083792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:47.083382 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw" event={"ID":"c00f1c3e-401f-461c-811f-71a6cf5ba1eb","Type":"ContainerDied","Data":"a4fba457673eeb65f16325e5459dfc05e78e90875457c0a0a68303fbdcc4f51e"} Oct 31 03:35:47.083792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:47.083436 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4fba457673eeb65f16325e5459dfc05e78e90875457c0a0a68303fbdcc4f51e" Oct 31 03:35:47.720200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:47.720159 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:35:49.636983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:49.636940 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:49.650454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:49.650425 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:50.118793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:50.118764 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:35:50.304206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:50.304164 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:51.248048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248016 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc"] Oct 31 03:35:51.248531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248514 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c00f1c3e-401f-461c-811f-71a6cf5ba1eb" containerName="revoke-keys" Oct 31 03:35:51.248576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248532 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00f1c3e-401f-461c-811f-71a6cf5ba1eb" containerName="revoke-keys" Oct 31 03:35:51.248576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248553 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b73b3f32-3340-4e62-9bac-20284e921d61" containerName="revoke-keys" Oct 31 03:35:51.248576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248558 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73b3f32-3340-4e62-9bac-20284e921d61" containerName="revoke-keys" Oct 31 03:35:51.248662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248625 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00f1c3e-401f-461c-811f-71a6cf5ba1eb" containerName="revoke-keys" Oct 31 03:35:51.248662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.248636 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73b3f32-3340-4e62-9bac-20284e921d61" containerName="revoke-keys" Oct 31 03:35:51.251916 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.251891 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.324898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.324865 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.325073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.324933 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mns25\" (UniqueName: \"kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.325073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.325058 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.356585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.356548 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:51.356585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.356547 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:51.356585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.356547 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:51.356928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.356678 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:35:51.359719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.359693 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc"] Oct 31 03:35:51.426413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.426380 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.426598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.426503 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.426598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.426568 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mns25\" (UniqueName: \"kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.427212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.427188 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.429029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.429006 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.487908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.487872 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mns25\" (UniqueName: \"kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25\") pod \"maas-api-revoke-sub-f0e28563-n6lpc\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.565640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.565554 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:51.942428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:51.942394 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc"] Oct 31 03:35:51.943991 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:51.943961 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dd0af4_353e_4aa9_af17_43a22de131a9.slice/crio-ab8863f4cf60c42896fbaf0e6f57b55324099c552697cc58803fe2210a21333d WatchSource:0}: Error finding container ab8863f4cf60c42896fbaf0e6f57b55324099c552697cc58803fe2210a21333d: Status 404 returned error can't find the container with id ab8863f4cf60c42896fbaf0e6f57b55324099c552697cc58803fe2210a21333d Oct 31 03:35:52.125389 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.125360 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" event={"ID":"06dd0af4-353e-4aa9-af17-43a22de131a9","Type":"ContainerStarted","Data":"ab8863f4cf60c42896fbaf0e6f57b55324099c552697cc58803fe2210a21333d"} Oct 31 03:35:52.874648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.874606 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph"] Oct 31 03:35:52.878555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.878531 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945599 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945643 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945670 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945896 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945796 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945896 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945840 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945929 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.945975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945951 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77gs\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.946069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.945982 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.946069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.946055 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:52.983609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:52.983573 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph"] Oct 31 03:35:53.046795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.046704 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.046795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.046753 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77gs\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.046929 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047009 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047112 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047156 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047191 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047234 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047195 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047301 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047346 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047464 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047543 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.047739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.047718 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.049626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.049601 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.049953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.049938 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.108629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.108594 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.108790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.108763 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77gs\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.140524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.140483 4488 generic.go:356] "Generic (PLEG): container finished" podID="06dd0af4-353e-4aa9-af17-43a22de131a9" containerID="364c2acc3b1b840b1466f9cbc694f0e656173e2e43bfe51be061b9f558e1ffb4" exitCode=0 Oct 31 03:35:53.140693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.140521 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" event={"ID":"06dd0af4-353e-4aa9-af17-43a22de131a9","Type":"ContainerDied","Data":"364c2acc3b1b840b1466f9cbc694f0e656173e2e43bfe51be061b9f558e1ffb4"} Oct 31 03:35:53.191349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.191318 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:53.208465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.208436 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj"] Oct 31 03:35:53.212403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.212379 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.274450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.274399 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj"] Oct 31 03:35:53.350701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.350652 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.350888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.350719 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.350888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.350789 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.350888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.350832 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.350888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.350869 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6vj\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.351070 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.350963 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.351070 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.351006 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.351196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.351120 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.351196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.351163 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.452367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452335 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.452540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452387 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.452540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452499 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6vj\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.452540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452538 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.452727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452609 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452929 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.452994 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453236 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453238 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453336 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453339 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453340 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453491 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.453996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.453661 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.455026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.455002 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.455374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.455352 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.514450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.514420 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.514690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.514668 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6vj\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.527451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.527426 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:53.626569 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:53.626541 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0e602d_d0bf_49fc_b13a_ea527237af0d.slice/crio-6fe725a25fec9d2ef22c22e285002d50b92d9da6c0bc7023ace39c168805add6 WatchSource:0}: Error finding container 6fe725a25fec9d2ef22c22e285002d50b92d9da6c0bc7023ace39c168805add6: Status 404 returned error can't find the container with id 6fe725a25fec9d2ef22c22e285002d50b92d9da6c0bc7023ace39c168805add6 Oct 31 03:35:53.626718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.626681 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph"] Oct 31 03:35:53.629284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.629243 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:35:53.629365 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.629337 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:35:53.629408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.629384 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:35:53.913687 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:35:53.913649 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c60e57_0008_4c21_8157_ebaa654a3fbf.slice/crio-8438d4590ce6376e2fb46fef4102bc0e8e2023f7b5c8dac60e723ae8f329e66c WatchSource:0}: Error finding container 8438d4590ce6376e2fb46fef4102bc0e8e2023f7b5c8dac60e723ae8f329e66c: Status 404 returned error can't find the container with id 8438d4590ce6376e2fb46fef4102bc0e8e2023f7b5c8dac60e723ae8f329e66c Oct 31 03:35:53.914181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.914111 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj"] Oct 31 03:35:53.916700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.916661 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:35:53.916779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.916748 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:35:53.916821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:53.916793 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:35:54.148905 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.148863 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" event={"ID":"a7c60e57-0008-4c21-8157-ebaa654a3fbf","Type":"ContainerStarted","Data":"018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea"} Oct 31 03:35:54.149141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.148923 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" event={"ID":"a7c60e57-0008-4c21-8157-ebaa654a3fbf","Type":"ContainerStarted","Data":"8438d4590ce6376e2fb46fef4102bc0e8e2023f7b5c8dac60e723ae8f329e66c"} Oct 31 03:35:54.150745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.150715 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" event={"ID":"3b0e602d-d0bf-49fc-b13a-ea527237af0d","Type":"ContainerStarted","Data":"56a59a33bb8497ec045128cc24df4393dc4a8dff9df9d377f2581b4c7519771c"} Oct 31 03:35:54.150841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.150769 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" event={"ID":"3b0e602d-d0bf-49fc-b13a-ea527237af0d","Type":"ContainerStarted","Data":"6fe725a25fec9d2ef22c22e285002d50b92d9da6c0bc7023ace39c168805add6"} Oct 31 03:35:54.192595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.192505 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:35:54.194401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.194374 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:35:54.194481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.194434 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:35:54.271774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.271712 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podStartSLOduration=1.271697661 podStartE2EDuration="1.271697661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:54.268759272 +0000 UTC m=+982.065477544" watchObservedRunningTime="2026-10-31 03:35:54.271697661 +0000 UTC m=+982.068415919" Oct 31 03:35:54.331409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.331370 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:54.401059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.400991 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podStartSLOduration=2.400972081 podStartE2EDuration="2.400972081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:54.399423364 +0000 UTC m=+982.196141639" watchObservedRunningTime="2026-10-31 03:35:54.400972081 +0000 UTC m=+982.197690341" Oct 31 03:35:54.467449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.467409 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca\") pod \"06dd0af4-353e-4aa9-af17-43a22de131a9\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " Oct 31 03:35:54.467645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.467478 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca\") pod \"06dd0af4-353e-4aa9-af17-43a22de131a9\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " Oct 31 03:35:54.467758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.467674 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25\" (UniqueName: \"kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25\") pod \"06dd0af4-353e-4aa9-af17-43a22de131a9\" (UID: \"06dd0af4-353e-4aa9-af17-43a22de131a9\") " Oct 31 03:35:54.467879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.467851 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca" pod "06dd0af4-353e-4aa9-af17-43a22de131a9" (UID: "06dd0af4-353e-4aa9-af17-43a22de131a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:54.468845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.468268 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:54.472145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.470652 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25" pod "06dd0af4-353e-4aa9-af17-43a22de131a9" (UID: "06dd0af4-353e-4aa9-af17-43a22de131a9"). InnerVolumeSpecName "kube-api-access-mns25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:54.474764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.473677 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca" pod "06dd0af4-353e-4aa9-af17-43a22de131a9" (UID: "06dd0af4-353e-4aa9-af17-43a22de131a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:54.537798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.537764 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:35:54.561348 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.545316 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:35:54.561348 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.545376 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:35:54.569504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.569461 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd0af4-353e-4aa9-af17-43a22de131a9-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:54.569504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:54.569498 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mns25\" (UniqueName: \"kubernetes.io/projected/06dd0af4-353e-4aa9-af17-43a22de131a9-kube-api-access-mns25\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:35:55.157281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.144846 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.178135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.177368 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" Oct 31 03:35:55.182127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.180075 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc" event={"ID":"06dd0af4-353e-4aa9-af17-43a22de131a9","Type":"ContainerDied","Data":"ab8863f4cf60c42896fbaf0e6f57b55324099c552697cc58803fe2210a21333d"} Oct 31 03:35:55.182352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.182297 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8863f4cf60c42896fbaf0e6f57b55324099c552697cc58803fe2210a21333d" Oct 31 03:35:55.192684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.192213 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:35:55.192684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.192269 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:35:55.557192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.556692 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:35:55.557192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:55.556773 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:35:56.192718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:56.192687 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:35:56.193122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:56.192734 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:35:56.537552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:56.531691 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:35:56.537552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:56.531779 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:35:57.192020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:57.191986 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:35:57.192197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:57.192060 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:35:57.529191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:57.528585 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:35:57.529191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:57.528656 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:35:58.192143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:58.192080 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:35:58.192299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:58.192160 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:35:58.533187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:58.528622 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:35:58.533187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:58.528689 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:35:59.192184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:59.191701 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:35:59.192184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:59.191748 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:35:59.536173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:59.535522 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:35:59.536173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:59.535586 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:35:59.984879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:35:59.984846 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:00.191995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:00.191952 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:36:00.192261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:00.192024 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 03:36:00.528043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:00.528008 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:36:00.528244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:00.528066 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 03:36:01.197463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.197425 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:36:01.197930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.197913 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:36:01.198566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.198545 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:36:01.532315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.532225 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:36:01.532654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.532633 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:36:01.533448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.533423 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:36:01.996101 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.996051 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj"] Oct 31 03:36:01.996538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.996523 4488 cpu_manager.go:414] "containerMap: removing container" podUID="06dd0af4-353e-4aa9-af17-43a22de131a9" containerName="revoke-keys" Oct 31 03:36:01.996597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.996540 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd0af4-353e-4aa9-af17-43a22de131a9" containerName="revoke-keys" Oct 31 03:36:01.996662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:01.996652 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd0af4-353e-4aa9-af17-43a22de131a9" containerName="revoke-keys" Oct 31 03:36:02.000245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.000229 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.055150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.055078 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:36:02.055405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.055266 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:36:02.055405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.055346 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:36:02.055726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.055702 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:36:02.059488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.059449 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj"] Oct 31 03:36:02.101656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.101617 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mvh\" (UniqueName: \"kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.101832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.101673 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.101880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.101845 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.202834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.202791 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.203363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.202894 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mvh\" (UniqueName: \"kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.203363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.202940 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.203742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.203716 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.205356 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.205334 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.264531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.264452 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mvh\" (UniqueName: \"kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh\") pod \"maas-api-revoke-sub-8439c8dc-jszkj\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.313515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.313480 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:02.723379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:02.723347 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj"] Oct 31 03:36:02.724305 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:36:02.724279 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1132c8_7514_48bd_bd3e_c266c5f36ad3.slice/crio-073ed35c39ecaaba615bed652e64417560e150192ff022efd4a3967b41096a26 WatchSource:0}: Error finding container 073ed35c39ecaaba615bed652e64417560e150192ff022efd4a3967b41096a26: Status 404 returned error can't find the container with id 073ed35c39ecaaba615bed652e64417560e150192ff022efd4a3967b41096a26 Oct 31 03:36:03.262961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:03.262923 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" event={"ID":"9b1132c8-7514-48bd-bd3e-c266c5f36ad3","Type":"ContainerStarted","Data":"693ca5b3172ecd93c4f99cf728e6ba7ccd811600ff4897c9f1f41bd77bb36167"} Oct 31 03:36:03.263402 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:03.262988 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" event={"ID":"9b1132c8-7514-48bd-bd3e-c266c5f36ad3","Type":"ContainerStarted","Data":"073ed35c39ecaaba615bed652e64417560e150192ff022efd4a3967b41096a26"} Oct 31 03:36:03.379320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:03.379269 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" podStartSLOduration=2.37924739 podStartE2EDuration="2.37924739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:03.378710234 +0000 UTC m=+991.175428495" watchObservedRunningTime="2026-10-31 03:36:03.37924739 +0000 UTC m=+991.175965651" Oct 31 03:36:04.270955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:04.270921 4488 generic.go:356] "Generic (PLEG): container finished" podID="9b1132c8-7514-48bd-bd3e-c266c5f36ad3" containerID="693ca5b3172ecd93c4f99cf728e6ba7ccd811600ff4897c9f1f41bd77bb36167" exitCode=0 Oct 31 03:36:04.271441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:04.271004 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" event={"ID":"9b1132c8-7514-48bd-bd3e-c266c5f36ad3","Type":"ContainerDied","Data":"693ca5b3172ecd93c4f99cf728e6ba7ccd811600ff4897c9f1f41bd77bb36167"} Oct 31 03:36:04.813153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:04.813080 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:05.429474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.429451 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:05.439852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.439830 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca\") pod \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " Oct 31 03:36:05.439965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.439869 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh\" (UniqueName: \"kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh\") pod \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " Oct 31 03:36:05.439965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.439894 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca\") pod \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\" (UID: \"9b1132c8-7514-48bd-bd3e-c266c5f36ad3\") " Oct 31 03:36:05.440265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.440241 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca" pod "9b1132c8-7514-48bd-bd3e-c266c5f36ad3" (UID: "9b1132c8-7514-48bd-bd3e-c266c5f36ad3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:05.440423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.440407 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:05.442056 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.442035 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca" pod "9b1132c8-7514-48bd-bd3e-c266c5f36ad3" (UID: "9b1132c8-7514-48bd-bd3e-c266c5f36ad3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:05.442056 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.442042 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh" pod "9b1132c8-7514-48bd-bd3e-c266c5f36ad3" (UID: "9b1132c8-7514-48bd-bd3e-c266c5f36ad3"). InnerVolumeSpecName "kube-api-access-s7mvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:05.541047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.540977 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7mvh\" (UniqueName: \"kubernetes.io/projected/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-kube-api-access-s7mvh\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:05.541047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:05.541003 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1132c8-7514-48bd-bd3e-c266c5f36ad3-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:06.284019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:06.283987 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" Oct 31 03:36:06.284201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:06.284016 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj" event={"ID":"9b1132c8-7514-48bd-bd3e-c266c5f36ad3","Type":"ContainerDied","Data":"073ed35c39ecaaba615bed652e64417560e150192ff022efd4a3967b41096a26"} Oct 31 03:36:06.284201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:06.284122 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="073ed35c39ecaaba615bed652e64417560e150192ff022efd4a3967b41096a26" Oct 31 03:36:06.588972 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:36:06.588941 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:36:09.604729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.604691 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8"] Oct 31 03:36:09.605255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.605226 4488 cpu_manager.go:414] "containerMap: removing container" podUID="9b1132c8-7514-48bd-bd3e-c266c5f36ad3" containerName="revoke-keys" Oct 31 03:36:09.605255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.605239 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1132c8-7514-48bd-bd3e-c266c5f36ad3" containerName="revoke-keys" Oct 31 03:36:09.605361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.605333 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1132c8-7514-48bd-bd3e-c266c5f36ad3" containerName="revoke-keys" Oct 31 03:36:09.609168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.609145 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.613238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.613210 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:09.726035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.725999 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6kn4\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726072 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726128 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726158 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726173 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726193 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726224 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726246 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.726644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.726361 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.758564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.758519 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8"] Oct 31 03:36:09.827665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827609 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.827869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827712 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.827869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827758 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6kn4\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.827869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827794 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.827869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827817 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.827869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827848 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.827979 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828025 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828078 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828250 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828420 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828467 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828551 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.828892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.828867 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.830792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.830778 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.831010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.830992 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.889747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.889675 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6kn4\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.889747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.889707 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:09.923001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:09.922968 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:10.400959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:10.400929 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8"] Oct 31 03:36:10.402266 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:36:10.402233 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55945c52_6050_46ea_98ad_3a561451409a.slice/crio-a647f5c94878a0657ab597f3e49933bac4aa0009ecf86d8e42fbdcbb3df5bc34 WatchSource:0}: Error finding container a647f5c94878a0657ab597f3e49933bac4aa0009ecf86d8e42fbdcbb3df5bc34: Status 404 returned error can't find the container with id a647f5c94878a0657ab597f3e49933bac4aa0009ecf86d8e42fbdcbb3df5bc34 Oct 31 03:36:10.405369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:10.405316 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:36:10.405466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:10.405414 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:36:10.405539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:10.405461 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:36:11.334380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:11.334341 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" event={"ID":"55945c52-6050-46ea-98ad-3a561451409a","Type":"ContainerStarted","Data":"bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa"} Oct 31 03:36:11.334878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:11.334410 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" event={"ID":"55945c52-6050-46ea-98ad-3a561451409a","Type":"ContainerStarted","Data":"a647f5c94878a0657ab597f3e49933bac4aa0009ecf86d8e42fbdcbb3df5bc34"} Oct 31 03:36:11.521113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:11.509838 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podStartSLOduration=2.509817152 podStartE2EDuration="2.509817152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:11.5044244 +0000 UTC m=+999.301142660" watchObservedRunningTime="2026-10-31 03:36:11.509817152 +0000 UTC m=+999.306535411" Oct 31 03:36:11.925252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:11.925163 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:11.929291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:11.929256 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:36:11.929495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:11.929326 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:36:12.929158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:12.927590 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:36:12.929158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:12.927657 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:36:13.929740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:13.929698 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:36:13.930336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:13.929760 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:36:14.538012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:14.537968 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:14.927038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:14.926568 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:36:14.927038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:14.926638 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:36:15.925172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:15.925123 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:36:15.925786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:15.925753 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:36:16.645433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:16.645391 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.825357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:16.825323 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.924140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:16.924011 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:36:16.924291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:16.924137 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:36:17.927754 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:17.927725 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:17.928204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:17.928119 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:17.928811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:17.928796 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:36:19.415042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:19.415013 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:21.900278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:21.900244 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:24.207197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:24.207159 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:28.970792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:28.970759 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:36:29.005560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:29.005529 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.098428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.098388 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6"] Oct 31 03:36:30.098867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.098701 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" containerID="cri-o://e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8" gracePeriod=30 Oct 31 03:36:30.148610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.148571 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs"] Oct 31 03:36:30.153810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.153776 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.204966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.204934 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrfb\" (UniqueName: \"kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.204966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.204973 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.205236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.205064 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.205236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.205135 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.205236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.205223 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.205349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.205251 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.208948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.208907 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"kube-root-ca.crt\"" Oct 31 03:36:30.208948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.208915 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"openshift-service-ca.crt\"" Oct 31 03:36:30.209224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.209012 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"distinct-w4-096812-kserve-self-signed-certs\"" Oct 31 03:36:30.209224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.209039 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"default-dockercfg-jm4bw\"" Oct 31 03:36:30.212673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.212637 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs"] Oct 31 03:36:30.244720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.244683 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.307420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.307379 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrfb\" (UniqueName: \"kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.307692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.307429 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.307692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.307472 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.307692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.307495 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.307692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.307544 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.307692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.307572 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.308231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.308198 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.308487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.308402 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.308487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.308511 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.310801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.310777 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.312668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.312644 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.369283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.369212 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrfb\" (UniqueName: \"kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb\") pod \"distinct-w4-096812-kserve-8c4d8f7b7-dd8hs\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.471041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.471005 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:30.873625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:30.873596 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs"] Oct 31 03:36:30.874728 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:36:30.874700 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf22472fe_b5bb_4545_b3f4_1b589c55d349.slice/crio-959dc14b3ac9ac8f919d70a11d4383f47ed5adb02e65c9ccfec5bc7f00b29321 WatchSource:0}: Error finding container 959dc14b3ac9ac8f919d70a11d4383f47ed5adb02e65c9ccfec5bc7f00b29321: Status 404 returned error can't find the container with id 959dc14b3ac9ac8f919d70a11d4383f47ed5adb02e65c9ccfec5bc7f00b29321 Oct 31 03:36:31.474876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:31.474836 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" event={"ID":"f22472fe-b5bb-4545-b3f4-1b589c55d349","Type":"ContainerStarted","Data":"be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8"} Oct 31 03:36:31.475360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:31.474888 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" event={"ID":"f22472fe-b5bb-4545-b3f4-1b589c55d349","Type":"ContainerStarted","Data":"959dc14b3ac9ac8f919d70a11d4383f47ed5adb02e65c9ccfec5bc7f00b29321"} Oct 31 03:36:31.592979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:31.592924 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podStartSLOduration=1.592910571 podStartE2EDuration="1.592910571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:31.591886654 +0000 UTC m=+1019.388604914" watchObservedRunningTime="2026-10-31 03:36:31.592910571 +0000 UTC m=+1019.389628830" Oct 31 03:36:32.471474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:32.471444 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:33.915327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:33.915290 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:34.079764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:34.079724 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:35.377968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.377900 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:36:35.472436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472396 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472487 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472513 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472546 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472575 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472641 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472719 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472750 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472768 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:35.472915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472792 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data\") pod \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\" (UID: \"f5cf0fca-bad8-48f4-95fc-34c3cee38534\") " Oct 31 03:36:35.472915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.472875 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:35.473273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.473250 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:35.473499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.473412 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.473499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.473437 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.473499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.473433 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:35.473499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.473451 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.473499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.473410 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:35.475189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.475129 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:36:35.475299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.475206 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:35.475528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.475496 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:35.475626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.475606 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf" pod "f5cf0fca-bad8-48f4-95fc-34c3cee38534" (UID: "f5cf0fca-bad8-48f4-95fc-34c3cee38534"). InnerVolumeSpecName "kube-api-access-vqbmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:35.499970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.499919 4488 generic.go:356] "Generic (PLEG): container finished" podID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerID="e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8" exitCode=0 Oct 31 03:36:35.500189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.500008 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" Oct 31 03:36:35.500189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.500003 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" event={"ID":"f5cf0fca-bad8-48f4-95fc-34c3cee38534","Type":"ContainerDied","Data":"e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8"} Oct 31 03:36:35.500189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.500173 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6" event={"ID":"f5cf0fca-bad8-48f4-95fc-34c3cee38534","Type":"ContainerDied","Data":"c85c370eae93806c9662b9177083266c2d8dc8cad71d8f15f1f9486f62b33a67"} Oct 31 03:36:35.500351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.500200 4488 scope.go:122] "RemoveContainer" containerID="e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8" Oct 31 03:36:35.510529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.510509 4488 scope.go:122] "RemoveContainer" containerID="e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8" Oct 31 03:36:35.510861 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:36:35.510836 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8\": container with ID starting with e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8 not found: ID does not exist" containerID="e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8" Oct 31 03:36:35.510914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.510867 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8"} err="failed to get container status \"e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8\": rpc error: code = NotFound desc = could not find container \"e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8\": container with ID starting with e3a73b04d82554ffe6ab218fbd83dfbfbd78b1a3c55ac3c5ad8ca59d6c681eb8 not found: ID does not exist" Oct 31 03:36:35.574200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.574169 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.574351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.574217 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqbmf\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-kube-api-access-vqbmf\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.574351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.574229 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.574351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.574238 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.574351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.574248 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.574351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.574256 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5cf0fca-bad8-48f4-95fc-34c3cee38534-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:36:35.667914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.667822 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6"] Oct 31 03:36:35.672594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:35.672565 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bt2j6"] Oct 31 03:36:36.316683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:36.316648 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" path="/var/lib/kubelet/pods/f5cf0fca-bad8-48f4-95fc-34c3cee38534/volumes" Oct 31 03:36:36.592265 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:36:36.592223 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:36:37.193348 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.193305 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv"] Oct 31 03:36:37.193825 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.193807 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" Oct 31 03:36:37.193935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.193828 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" Oct 31 03:36:37.193998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.193937 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5cf0fca-bad8-48f4-95fc-34c3cee38534" containerName="istio-proxy" Oct 31 03:36:37.198672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.198651 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.256827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.256793 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv"] Oct 31 03:36:37.279725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.279684 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"distinct-w1-e00ad3-kserve-self-signed-certs\"" Oct 31 03:36:37.292373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.292332 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.292556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.292390 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.292556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.292455 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzrj\" (UniqueName: \"kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.292644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.292564 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.292693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.292673 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.292811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.292794 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.308304 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.308271 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:37.393793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.393669 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.393793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.393735 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.393793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.393771 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.393793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.393789 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzrj\" (UniqueName: \"kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.393860 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.393910 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.394178 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.394255 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.394301 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.396462 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.411871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.396790 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.455793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.455714 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzrj\" (UniqueName: \"kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj\") pod \"distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.510491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.510458 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:37.887236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:37.887203 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv"] Oct 31 03:36:37.889189 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:36:37.889161 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd27142dd_64fc_47fc_b2fa_a1105267a78d.slice/crio-28e5e905682c0240322dbcb88fc7351d61f749d117ee50a2a6c7ce4988450823 WatchSource:0}: Error finding container 28e5e905682c0240322dbcb88fc7351d61f749d117ee50a2a6c7ce4988450823: Status 404 returned error can't find the container with id 28e5e905682c0240322dbcb88fc7351d61f749d117ee50a2a6c7ce4988450823 Oct 31 03:36:38.523873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:38.523834 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" event={"ID":"d27142dd-64fc-47fc-b2fa-a1105267a78d","Type":"ContainerStarted","Data":"aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a"} Oct 31 03:36:38.524111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:38.523879 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" event={"ID":"d27142dd-64fc-47fc-b2fa-a1105267a78d","Type":"ContainerStarted","Data":"28e5e905682c0240322dbcb88fc7351d61f749d117ee50a2a6c7ce4988450823"} Oct 31 03:36:38.642037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:38.641982 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podStartSLOduration=1.641966037 podStartE2EDuration="1.641966037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:38.640621433 +0000 UTC m=+1026.437339694" watchObservedRunningTime="2026-10-31 03:36:38.641966037 +0000 UTC m=+1026.438684315" Oct 31 03:36:38.688993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:38.688962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:39.511104 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:39.511062 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:40.471272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:40.471237 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:40.483744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:40.483712 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:40.549289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:40.549262 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:36:43.531165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:43.531125 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:46.428745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.428712 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7"] Oct 31 03:36:46.432712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.432692 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.494718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.494684 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7"] Oct 31 03:36:46.501761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.501732 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.501761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.501763 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.501992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.501797 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.501992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.501912 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsm7\" (UniqueName: \"kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.501992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.501961 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.502187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.502067 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.530641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.530604 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:46.540800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.540776 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"distinct-w5-14cb5a-kserve-self-signed-certs\"" Oct 31 03:36:46.603424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603388 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.603623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603433 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.603623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603582 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.603745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603636 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsm7\" (UniqueName: \"kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.603745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603701 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.603852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603748 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.603921 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603873 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.604022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.603999 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.604200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.604170 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.606212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.606159 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.606800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.606778 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.665512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.665473 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsm7\" (UniqueName: \"kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7\") pod \"distinct-w5-14cb5a-kserve-6459776446-q4wh7\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:46.745860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:46.745782 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:47.139769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.139743 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7"] Oct 31 03:36:47.141078 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:36:47.141039 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04e96f43_d2e3_4015_bcda_9ae34c66b75b.slice/crio-3adf23777046d55cca529ef5e9de9977f6afbf924db3a2b9f10aeaa731111ba2 WatchSource:0}: Error finding container 3adf23777046d55cca529ef5e9de9977f6afbf924db3a2b9f10aeaa731111ba2: Status 404 returned error can't find the container with id 3adf23777046d55cca529ef5e9de9977f6afbf924db3a2b9f10aeaa731111ba2 Oct 31 03:36:47.510915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.510820 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:47.523703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.523673 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:47.590589 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.590540 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" event={"ID":"04e96f43-d2e3-4015-bcda-9ae34c66b75b","Type":"ContainerStarted","Data":"1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4"} Oct 31 03:36:47.590589 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.590587 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" event={"ID":"04e96f43-d2e3-4015-bcda-9ae34c66b75b","Type":"ContainerStarted","Data":"3adf23777046d55cca529ef5e9de9977f6afbf924db3a2b9f10aeaa731111ba2"} Oct 31 03:36:47.601853 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.601827 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:36:47.746250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.746223 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:47.760860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:47.760802 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podStartSLOduration=1.760789234 podStartE2EDuration="1.760789234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:47.757383436 +0000 UTC m=+1035.554101693" watchObservedRunningTime="2026-10-31 03:36:47.760789234 +0000 UTC m=+1035.557507492" Oct 31 03:36:48.325982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:48.325951 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:53.100251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:53.100180 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:55.326432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.326401 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt"] Oct 31 03:36:55.330327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.330311 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.383055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.383025 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-909365-openshift-default-dockercfg-rxcsh\"" Oct 31 03:36:55.436646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.436615 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt"] Oct 31 03:36:55.506661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.506625 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.506859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.506753 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.506859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.506830 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.506983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.506947 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.507034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.507003 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.507121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.507067 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpvn\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.507183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.507168 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.507245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.507192 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.507245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.507232 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.608642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.608607 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.608870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.608686 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.608870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.608723 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.608870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.608743 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpvn\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.608902 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.608959 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609000 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609123 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609132 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609170 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609233 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609235 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609419 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.609818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.609787 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.611497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.611479 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.611647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.611630 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.670124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.670057 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.670273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.670213 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpvn\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:55.944475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:55.944397 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:56.378490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.378456 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt"] Oct 31 03:36:56.415304 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:36:56.415269 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbc629c_1eb1_423f_bd9d_37d7f2cb8c22.slice/crio-f371485d33220f88861cd5843fd2bc8f2a7f7399a702c3032ea24c7a0fffa458 WatchSource:0}: Error finding container f371485d33220f88861cd5843fd2bc8f2a7f7399a702c3032ea24c7a0fffa458: Status 404 returned error can't find the container with id f371485d33220f88861cd5843fd2bc8f2a7f7399a702c3032ea24c7a0fffa458 Oct 31 03:36:56.418141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.418108 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:36:56.418242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.418178 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:36:56.418242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.418212 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:36:56.652652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.652565 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" event={"ID":"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22","Type":"ContainerStarted","Data":"4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22"} Oct 31 03:36:56.652652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.652624 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" event={"ID":"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22","Type":"ContainerStarted","Data":"f371485d33220f88861cd5843fd2bc8f2a7f7399a702c3032ea24c7a0fffa458"} Oct 31 03:36:56.746354 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.746325 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:56.759429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.759401 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:56.774712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.774479 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podStartSLOduration=1.7744620709999999 podStartE2EDuration="1.774462071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:56.774070946 +0000 UTC m=+1044.570789205" watchObservedRunningTime="2026-10-31 03:36:56.774462071 +0000 UTC m=+1044.571180330" Oct 31 03:36:56.945126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:56.945012 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:36:57.670549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:57.670519 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:36:57.878274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:57.878240 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:57.946880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:57.946791 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:36:57.946880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:57.946844 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:36:58.945748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:58.945708 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:36:58.946279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:58.945766 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:36:59.831368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:59.831333 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:59.945263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:59.945235 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:36:59.945426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:36:59.945289 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:37:00.944879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:00.944839 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:37:00.945354 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:00.944899 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:37:01.945737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:01.945693 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:37:01.946180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:01.945758 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:37:02.476596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:02.476552 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:37:02.794642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:02.794561 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:02.945941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:02.945462 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:37:02.945941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:02.945523 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:37:03.211920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:03.211874 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:03.968474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:03.968438 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:37:03.970575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:03.970552 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:37:03.982322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:03.982291 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:37:06.589625 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:37:06.589582 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:37:07.620967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:07.620919 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:12.550783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:12.550742 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:12.991561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:12.991528 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:13.155115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:13.155060 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:17.532374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:17.532329 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:22.248974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:22.248940 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:22.402508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:22.402474 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:23.944564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:23.944526 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6"] Oct 31 03:37:23.948573 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:23.948550 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.054934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.054904 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.054947 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7p5\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.054988 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.055205 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.055320 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.055395 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.055459 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.055565 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.055627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.055614 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.156725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.156689 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.156935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.156742 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.156935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.156823 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.156935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.156846 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7p5\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.156935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.156874 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.156935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.156932 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157010 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157035 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157056 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157414 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157513 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157532 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157692 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.157779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.157708 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.159513 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.159487 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.159613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.159580 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.166725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.166684 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6"] Oct 31 03:37:24.218932 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.218858 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.219127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.218968 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7p5\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.263189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.263153 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:24.699418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.699387 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6"] Oct 31 03:37:24.700808 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:37:24.700781 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2ae28f_266b_4f09_a723_d4baa8e887cf.slice/crio-a58b91f6f250bbe690aeccdef5acb80d32407d4511501424f32f3b83a642324e WatchSource:0}: Error finding container a58b91f6f250bbe690aeccdef5acb80d32407d4511501424f32f3b83a642324e: Status 404 returned error can't find the container with id a58b91f6f250bbe690aeccdef5acb80d32407d4511501424f32f3b83a642324e Oct 31 03:37:24.703396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.703359 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:37:24.703488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.703444 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:37:24.703488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.703482 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:37:24.898477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.898437 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" event={"ID":"2f2ae28f-266b-4f09-a723-d4baa8e887cf","Type":"ContainerStarted","Data":"bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9"} Oct 31 03:37:24.898665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:24.898506 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" event={"ID":"2f2ae28f-266b-4f09-a723-d4baa8e887cf","Type":"ContainerStarted","Data":"a58b91f6f250bbe690aeccdef5acb80d32407d4511501424f32f3b83a642324e"} Oct 31 03:37:25.020564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:25.020445 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podStartSLOduration=2.020425165 podStartE2EDuration="2.020425165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:25.019375865 +0000 UTC m=+1072.816094127" watchObservedRunningTime="2026-10-31 03:37:25.020425165 +0000 UTC m=+1072.817143426" Oct 31 03:37:25.263844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:25.263803 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:25.265312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:25.265284 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:25.265459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:25.265342 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:26.263634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:26.263590 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:26.264117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:26.263658 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:27.264292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:27.264254 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:27.264697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:27.264313 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:27.426115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:27.417468 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:28.264301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:28.264264 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:28.264728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:28.264321 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:29.263576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:29.263542 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:29.263762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:29.263603 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:30.263736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:30.263706 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:30.264161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:30.263774 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:31.264064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:31.264024 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:37:31.264506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:31.264135 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 03:37:32.269066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:32.269035 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:32.269614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:32.269598 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:32.270372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:32.270351 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:37:32.654439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:32.654384 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:34.070546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:34.070505 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:37:36.589673 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:37:36.589634 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:37:37.535998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.535895 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:37.814435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.814346 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p"] Oct 31 03:37:37.823647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.823615 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.854636 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.854596 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.854834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.854782 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.854929 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.854906 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.854966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.854947 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.855133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.855109 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.855191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.855176 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchjj\" (UniqueName: \"kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.876920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.876885 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p"] Oct 31 03:37:37.914556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.914516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:37.922727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.922708 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-dec4a0\"/\"distinct-2-w2-dec4a0-kserve-self-signed-certs\"" Oct 31 03:37:37.956076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956042 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.956343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956163 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.956343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956195 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.956343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956279 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.956519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956414 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchjj\" (UniqueName: \"kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.956519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956479 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.956624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.956593 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.957269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.957244 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.957482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.957335 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.959866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.959845 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:37.960208 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:37.960191 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:38.018180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:38.018146 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchjj\" (UniqueName: \"kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj\") pod \"distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:38.136965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:38.136928 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:38.532245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:38.532201 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p"] Oct 31 03:37:38.533132 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:37:38.533075 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70af8e7a_c2a3_4fe5_a812_cf3ad41562ab.slice/crio-c946c5c5982bcf3fde795c4c999b7540bc2558efb8dfa891a56344c88ba613ca WatchSource:0}: Error finding container c946c5c5982bcf3fde795c4c999b7540bc2558efb8dfa891a56344c88ba613ca: Status 404 returned error can't find the container with id c946c5c5982bcf3fde795c4c999b7540bc2558efb8dfa891a56344c88ba613ca Oct 31 03:37:39.014549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:39.014507 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" event={"ID":"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab","Type":"ContainerStarted","Data":"2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80"} Oct 31 03:37:39.014549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:39.014561 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" event={"ID":"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab","Type":"ContainerStarted","Data":"c946c5c5982bcf3fde795c4c999b7540bc2558efb8dfa891a56344c88ba613ca"} Oct 31 03:37:39.132651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:39.132597 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podStartSLOduration=2.132575992 podStartE2EDuration="2.132575992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:39.132245813 +0000 UTC m=+1086.928964083" watchObservedRunningTime="2026-10-31 03:37:39.132575992 +0000 UTC m=+1086.929294252" Oct 31 03:37:39.137956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:39.137925 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:42.316397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:42.316358 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:43.071860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:43.071823 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:47.125717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:47.125670 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:48.137204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:48.137167 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:48.150996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:48.150970 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:49.089369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:49.089342 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:37:52.018532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:52.018499 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:56.826981 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:37:56.826948 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:01.609497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:01.609462 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:06.400208 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:06.400170 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:06.589830 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:38:06.589798 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:38:06.795241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:06.795156 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:10.264709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.264674 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl"] Oct 31 03:38:10.268907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.268889 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.334508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.334473 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:38:10.354471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354440 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354477 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354508 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354619 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354676 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjqt\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354719 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354749 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354815 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.354867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.354853 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.367358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.367320 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl"] Oct 31 03:38:10.456181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456140 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjqt\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456193 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456235 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456319 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456386 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456480 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456518 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456566 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456608 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456669 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.456836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456705 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.457047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456994 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.457366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.456999 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.457447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.457421 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.459146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.459119 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.459582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.459558 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.518337 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.518263 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.518337 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.518327 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjqt\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:10.583340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:10.583298 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:11.017855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.017826 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl"] Oct 31 03:38:11.018983 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:38:11.018960 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2f9169_43a4_4aa2_920f_33774916eb13.slice/crio-ee09bc4db61cf2c8a8f53834a7edd22982eff77beb925d62bc7b86e9c2826be6 WatchSource:0}: Error finding container ee09bc4db61cf2c8a8f53834a7edd22982eff77beb925d62bc7b86e9c2826be6: Status 404 returned error can't find the container with id ee09bc4db61cf2c8a8f53834a7edd22982eff77beb925d62bc7b86e9c2826be6 Oct 31 03:38:11.021532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.021495 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:38:11.021643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.021583 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:38:11.021643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.021629 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:38:11.233868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.233831 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" event={"ID":"3d2f9169-43a4-4aa2-920f-33774916eb13","Type":"ContainerStarted","Data":"9d5fec12ff09556a3e4e54dccf945ce4eb5ddac0439134a71163dd9337869899"} Oct 31 03:38:11.234059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.233897 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" event={"ID":"3d2f9169-43a4-4aa2-920f-33774916eb13","Type":"ContainerStarted","Data":"ee09bc4db61cf2c8a8f53834a7edd22982eff77beb925d62bc7b86e9c2826be6"} Oct 31 03:38:11.244258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.244213 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:11.358105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.358039 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podStartSLOduration=1.358023974 podStartE2EDuration="1.358023974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:11.354643932 +0000 UTC m=+1119.151362201" watchObservedRunningTime="2026-10-31 03:38:11.358023974 +0000 UTC m=+1119.154742232" Oct 31 03:38:11.584226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:11.584194 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:12.585032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:12.584996 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:38:12.585505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:12.585048 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:38:13.583714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:13.583682 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:38:13.583923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:13.583736 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:38:14.584209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:14.584168 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:38:14.584727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:14.584253 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:38:15.584392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:15.584355 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:38:15.584899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:15.584430 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:38:16.013764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:16.013666 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:16.584317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:16.584279 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:38:16.584518 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:16.584352 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:38:17.584735 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:17.584695 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:38:17.585191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:17.584772 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:38:18.588607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:18.588576 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:18.589022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:18.588936 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:18.589784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:18.589762 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:38:20.818864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:20.818831 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:22.593566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:22.593527 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:25.601751 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:25.601714 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:30.508998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:30.508962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:35.305288 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:35.305256 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:36.590110 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:38:36.590043 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:38:39.384031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:39.383985 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:38:40.103985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:40.103933 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:44.928854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:44.928822 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:49.058865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:49.058826 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng"] Oct 31 03:38:49.108888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:49.108853 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fb2b05-l87ng"] Oct 31 03:38:49.709255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:49.709227 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:50.319937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:50.319889 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806d6f58-965d-42c4-b62c-80d51fd523f7" path="/var/lib/kubelet/pods/806d6f58-965d-42c4-b62c-80d51fd523f7/volumes" Oct 31 03:38:50.448499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:50.448466 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:54.530040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:54.530006 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:55.920876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:55.920833 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f"] Oct 31 03:38:55.925023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:55.925002 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:55.979908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:55.979859 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:38:56.029679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.029640 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:38:56.029679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.029660 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:38:56.029990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.029734 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:38:56.033349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.033325 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f"] Oct 31 03:38:56.079578 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.079543 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm5t\" (UniqueName: \"kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.079796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.079683 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.079858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.079797 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.181462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.181368 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.181462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.181452 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.181709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.181532 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm5t\" (UniqueName: \"kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.182192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.182167 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.184021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.184000 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.243399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.243364 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm5t\" (UniqueName: \"kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t\") pod \"maas-api-revoke-sub-bd915c1f-g7c6f\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.539152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.539026 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:56.937130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:56.937079 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f"] Oct 31 03:38:56.938261 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:38:56.938235 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18799e14_97c1_4493_beea_610b5cac08c0.slice/crio-d37b02184764fa9260629f139c224724c1ea2aa5fa896ba67a18d6974310c61e WatchSource:0}: Error finding container d37b02184764fa9260629f139c224724c1ea2aa5fa896ba67a18d6974310c61e: Status 404 returned error can't find the container with id d37b02184764fa9260629f139c224724c1ea2aa5fa896ba67a18d6974310c61e Oct 31 03:38:57.537274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:57.537157 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" event={"ID":"18799e14-97c1-4493-beea-610b5cac08c0","Type":"ContainerStarted","Data":"2e91d71ebd87979c9492a83a261f31ba5372b4257581d5bf716f5407ef99e0b0"} Oct 31 03:38:57.537274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:57.537259 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" event={"ID":"18799e14-97c1-4493-beea-610b5cac08c0","Type":"ContainerStarted","Data":"d37b02184764fa9260629f139c224724c1ea2aa5fa896ba67a18d6974310c61e"} Oct 31 03:38:57.655648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:57.655584 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" podStartSLOduration=2.6555660210000003 podStartE2EDuration="2.655566021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:57.655242395 +0000 UTC m=+1165.451960652" watchObservedRunningTime="2026-10-31 03:38:57.655566021 +0000 UTC m=+1165.452284280" Oct 31 03:38:58.546911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:58.546864 4488 generic.go:356] "Generic (PLEG): container finished" podID="18799e14-97c1-4493-beea-610b5cac08c0" containerID="2e91d71ebd87979c9492a83a261f31ba5372b4257581d5bf716f5407ef99e0b0" exitCode=0 Oct 31 03:38:58.547435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:58.546951 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" event={"ID":"18799e14-97c1-4493-beea-610b5cac08c0","Type":"ContainerDied","Data":"2e91d71ebd87979c9492a83a261f31ba5372b4257581d5bf716f5407ef99e0b0"} Oct 31 03:38:59.444437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.444402 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:59.705252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.705187 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:38:59.826985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.826951 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca\") pod \"18799e14-97c1-4493-beea-610b5cac08c0\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " Oct 31 03:38:59.827196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.827053 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca\") pod \"18799e14-97c1-4493-beea-610b5cac08c0\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " Oct 31 03:38:59.827259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.827226 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t\" (UniqueName: \"kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t\") pod \"18799e14-97c1-4493-beea-610b5cac08c0\" (UID: \"18799e14-97c1-4493-beea-610b5cac08c0\") " Oct 31 03:38:59.827458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.827426 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca" pod "18799e14-97c1-4493-beea-610b5cac08c0" (UID: "18799e14-97c1-4493-beea-610b5cac08c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:59.827650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.827634 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:38:59.829520 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.829480 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t" pod "18799e14-97c1-4493-beea-610b5cac08c0" (UID: "18799e14-97c1-4493-beea-610b5cac08c0"). InnerVolumeSpecName "kube-api-access-xrm5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:59.829845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.829822 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca" pod "18799e14-97c1-4493-beea-610b5cac08c0" (UID: "18799e14-97c1-4493-beea-610b5cac08c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:59.929187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.929152 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18799e14-97c1-4493-beea-610b5cac08c0-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:38:59.929187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:38:59.929183 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrm5t\" (UniqueName: \"kubernetes.io/projected/18799e14-97c1-4493-beea-610b5cac08c0-kube-api-access-xrm5t\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:00.562521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:00.562497 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" Oct 31 03:39:00.562696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:00.562487 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f" event={"ID":"18799e14-97c1-4493-beea-610b5cac08c0","Type":"ContainerDied","Data":"d37b02184764fa9260629f139c224724c1ea2aa5fa896ba67a18d6974310c61e"} Oct 31 03:39:00.562696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:00.562616 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37b02184764fa9260629f139c224724c1ea2aa5fa896ba67a18d6974310c61e" Oct 31 03:39:01.057180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:01.057144 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5"] Oct 31 03:39:01.059487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:01.059450 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246644c7-q5wd5"] Oct 31 03:39:02.318477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:02.318443 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d2b383-7f47-42a0-af4a-b7daf1d37e1d" path="/var/lib/kubelet/pods/f9d2b383-7f47-42a0-af4a-b7daf1d37e1d/volumes" Oct 31 03:39:03.108211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:03.108177 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:04.296212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:04.296177 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:06.589739 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:39:06.589700 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:39:09.211040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:09.211009 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:09.524573 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:09.524481 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:39:13.519745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.519709 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts"] Oct 31 03:39:13.520268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.520252 4488 cpu_manager.go:414] "containerMap: removing container" podUID="18799e14-97c1-4493-beea-610b5cac08c0" containerName="revoke-keys" Oct 31 03:39:13.520268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.520268 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18799e14-97c1-4493-beea-610b5cac08c0" containerName="revoke-keys" Oct 31 03:39:13.520386 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.520376 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="18799e14-97c1-4493-beea-610b5cac08c0" containerName="revoke-keys" Oct 31 03:39:13.524625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.524607 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.611026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.610982 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.611026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.611030 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.611326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.611182 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.611326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.611242 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.611326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.611271 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.611524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.611494 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ph6\" (UniqueName: \"kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.619340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.619307 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:13.628351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.628324 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"distinct-2-w5-14cb5a-kserve-self-signed-certs\"" Oct 31 03:39:13.631616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.631593 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts"] Oct 31 03:39:13.712809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.712773 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ph6\" (UniqueName: \"kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.712887 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.712914 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.712947 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.713029 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.713057 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.713466 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.713563 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.713618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.713572 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.715616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.715590 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.715816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.715798 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.774691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.774611 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ph6\" (UniqueName: \"kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6\") pod \"distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:13.837483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:13.837443 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:14.181165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:14.181062 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:14.262026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:14.262001 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts"] Oct 31 03:39:14.263695 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:14.263666 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d5d2cf4_f779_401f_986c_ed2e89685787.slice/crio-d017d20e90c0138d419f086f685c76172e6b8fc6e72c7078e4615789bc3a5782 WatchSource:0}: Error finding container d017d20e90c0138d419f086f685c76172e6b8fc6e72c7078e4615789bc3a5782: Status 404 returned error can't find the container with id d017d20e90c0138d419f086f685c76172e6b8fc6e72c7078e4615789bc3a5782 Oct 31 03:39:14.665214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:14.665160 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" event={"ID":"3d5d2cf4-f779-401f-986c-ed2e89685787","Type":"ContainerStarted","Data":"8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72"} Oct 31 03:39:14.665214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:14.665208 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" event={"ID":"3d5d2cf4-f779-401f-986c-ed2e89685787","Type":"ContainerStarted","Data":"d017d20e90c0138d419f086f685c76172e6b8fc6e72c7078e4615789bc3a5782"} Oct 31 03:39:14.786361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:14.786303 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podStartSLOduration=1.78628568 podStartE2EDuration="1.78628568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:14.783517546 +0000 UTC m=+1182.580235825" watchObservedRunningTime="2026-10-31 03:39:14.78628568 +0000 UTC m=+1182.583003937" Oct 31 03:39:14.837926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:14.837889 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:17.052641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:17.052607 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs"] Oct 31 03:39:17.102603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:17.102570 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4420ac5-8qpcs"] Oct 31 03:39:18.325048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:18.325009 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e8159e-7548-417e-b5a9-81373985ea62" path="/var/lib/kubelet/pods/d6e8159e-7548-417e-b5a9-81373985ea62/volumes" Oct 31 03:39:18.919420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:18.919383 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:20.622010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:20.621973 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:21.165227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.165192 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r"] Oct 31 03:39:21.169158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.169141 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.272958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.272920 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:39:21.272958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.272929 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:39:21.272958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.272956 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:39:21.273288 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.273168 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:39:21.276538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.276516 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r"] Oct 31 03:39:21.308032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.307996 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.308213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.308058 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22c5\" (UniqueName: \"kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.308259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.308234 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.410180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.410141 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.410392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.410373 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.410461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.410444 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22c5\" (UniqueName: \"kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.411182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.411152 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.413455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.413432 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.472121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.472015 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22c5\" (UniqueName: \"kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5\") pod \"maas-api-revoke-sub-e0109f1d-wsv4r\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.483534 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.483503 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:21.863579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:21.863547 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r"] Oct 31 03:39:21.864643 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:21.864615 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0c0e32_14f2_42ef_a04c_b6a32a8eb54d.slice/crio-63df057526e261b093baf3f3e5250c5caa4c211d3d1cc454e162af3915129281 WatchSource:0}: Error finding container 63df057526e261b093baf3f3e5250c5caa4c211d3d1cc454e162af3915129281: Status 404 returned error can't find the container with id 63df057526e261b093baf3f3e5250c5caa4c211d3d1cc454e162af3915129281 Oct 31 03:39:22.720586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:22.720552 4488 generic.go:356] "Generic (PLEG): container finished" podID="cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" containerID="92f2a2355a94231c09d1164a2e7b616a73b489948b6f55f55088e22354fe5d16" exitCode=0 Oct 31 03:39:22.720717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:22.720687 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" event={"ID":"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d","Type":"ContainerDied","Data":"92f2a2355a94231c09d1164a2e7b616a73b489948b6f55f55088e22354fe5d16"} Oct 31 03:39:22.720787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:22.720746 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" event={"ID":"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d","Type":"ContainerStarted","Data":"63df057526e261b093baf3f3e5250c5caa4c211d3d1cc454e162af3915129281"} Oct 31 03:39:23.063350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:23.063261 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:23.838921 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:23.838895 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:23.870210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:23.870179 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:23.910461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:23.910436 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:24.056797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.056707 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca\") pod \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " Oct 31 03:39:24.056984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.056868 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5\" (UniqueName: \"kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5\") pod \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " Oct 31 03:39:24.056984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.056915 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca\") pod \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\" (UID: \"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d\") " Oct 31 03:39:24.057378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.057347 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca" pod "cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" (UID: "cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:24.059174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.059151 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5" pod "cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" (UID: "cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d"). InnerVolumeSpecName "kube-api-access-c22c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:24.059247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.059147 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca" pod "cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" (UID: "cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:24.158801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.158768 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c22c5\" (UniqueName: \"kubernetes.io/projected/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-kube-api-access-c22c5\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:24.158801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.158796 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:24.158801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.158809 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:24.739522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.739491 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" Oct 31 03:39:24.739712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.739515 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r" event={"ID":"cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d","Type":"ContainerDied","Data":"63df057526e261b093baf3f3e5250c5caa4c211d3d1cc454e162af3915129281"} Oct 31 03:39:24.739712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.739571 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63df057526e261b093baf3f3e5250c5caa4c211d3d1cc454e162af3915129281" Oct 31 03:39:24.751189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.751163 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:24.836098 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.836066 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:24.991760 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:24.991682 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:25.871431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.871392 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49"] Oct 31 03:39:25.871935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.871903 4488 cpu_manager.go:414] "containerMap: removing container" podUID="cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" containerName="revoke-keys" Oct 31 03:39:25.871935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.871916 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" containerName="revoke-keys" Oct 31 03:39:25.872035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.872027 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" containerName="revoke-keys" Oct 31 03:39:25.875330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.875304 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.986769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.986723 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.986989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.986866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987014 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987064 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987215 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987312 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987400 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987376 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987470 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:25.987640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:25.987611 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7mz\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.088965 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089038 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089077 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089165 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089251 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7mz\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089285 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089331 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089431 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.089472 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.092802 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.093076 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.093747 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.094284 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49"] Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.094579 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.094643 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.094899 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.097551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.097475 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.152213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.152135 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.156188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.156161 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7mz\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.191210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.191174 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:26.653836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.653801 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49"] Oct 31 03:39:26.656127 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:26.656059 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a86723a_a51e_4e8d_b121_bc831aaf1456.slice/crio-44edd9bb868caa875586b8f4d2a655bc567ef161fcbed479544ed97ff0759b01 WatchSource:0}: Error finding container 44edd9bb868caa875586b8f4d2a655bc567ef161fcbed479544ed97ff0759b01: Status 404 returned error can't find the container with id 44edd9bb868caa875586b8f4d2a655bc567ef161fcbed479544ed97ff0759b01 Oct 31 03:39:26.659976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.659899 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:26.660156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.660025 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:26.660156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.660076 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:26.762974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.762923 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" event={"ID":"1a86723a-a51e-4e8d-b121-bc831aaf1456","Type":"ContainerStarted","Data":"6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4"} Oct 31 03:39:26.763191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:26.763005 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" event={"ID":"1a86723a-a51e-4e8d-b121-bc831aaf1456","Type":"ContainerStarted","Data":"44edd9bb868caa875586b8f4d2a655bc567ef161fcbed479544ed97ff0759b01"} Oct 31 03:39:27.192140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:27.191564 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:27.193684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:27.193660 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:27.193862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:27.193703 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:28.067230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.067173 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:28.198711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.198200 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:28.198711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.198279 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:28.390117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.387828 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.390117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.389112 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.446785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.446725 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.577335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.577290 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.578925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.578880 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.597255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.597206 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.611238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.611191 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.669885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.669773 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.786221 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:28.786144 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.010342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.010237 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:29.134625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.134569 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.139007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.138965 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.173820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.173765 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 31 03:39:29.214364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.214311 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:29.214984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.214949 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:29.276179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.226244 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.421212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.398335 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.421212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.398745 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:39:29.421212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.399905 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:39:29.472137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.459020 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.472137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.459519 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:39:29.548194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.524509 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podStartSLOduration=4.524475475 podStartE2EDuration="4.524475475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:26.912702395 +0000 UTC m=+1194.709420657" watchObservedRunningTime="2026-10-31 03:39:29.524475475 +0000 UTC m=+1197.321193734" Oct 31 03:39:29.551272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.551125 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.585968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.585926 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:39:29.586580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.586552 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:39:29.587066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.587045 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:39:29.587275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.587259 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:39:29.697259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.697211 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.749885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.747577 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:39:29.749885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.733242 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.749885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.736978 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.749885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.749063 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:39:29.830246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.828249 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:39:29.830246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.828734 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:29.830246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.829117 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:29.905844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:29.902722 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:39:30.064112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.062626 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:39:30.089448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.088732 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:39:30.089448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.088800 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:39:30.092114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.090490 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:39:30.092114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.090548 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:39:30.097343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.097317 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:39:30.191802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.191763 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:30.192068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:30.191918 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:31.192266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:31.192224 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:31.192792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:31.192294 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:32.191777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:32.191742 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:32.191986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:32.191795 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:33.191822 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:33.191788 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:33.192294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:33.191841 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:34.004441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:34.004411 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:34.192167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:34.192124 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:39:34.192570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:34.192196 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:39:35.197072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.197039 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:35.197555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.197364 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:35.198248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.198230 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:39:35.318568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.318529 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm"] Oct 31 03:39:35.324175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.324154 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.408491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.408451 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.408734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.408713 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvj7\" (UniqueName: \"kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.408919 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.408897 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.416811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.416770 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:39:35.416811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.416778 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:39:35.416811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.416811 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:39:35.417108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.416862 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:39:35.421934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.421908 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm"] Oct 31 03:39:35.513261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.513159 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.513432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.513333 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvj7\" (UniqueName: \"kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.513499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.513466 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.514755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.514730 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.523364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.522476 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.575508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.575477 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvj7\" (UniqueName: \"kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7\") pod \"maas-api-revoke-sub-4e5f1f43-bvwfm\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:35.638303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:35.638271 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:36.019280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.019248 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm"] Oct 31 03:39:36.020205 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:36.020169 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe99c841_f0fd_4ede_a3c5_3a9c60f317c0.slice/crio-7c7c1b113af7af1a50d4b3ec892d5c0ce99ad82f2b0dc012ba3464c489bcf77b WatchSource:0}: Error finding container 7c7c1b113af7af1a50d4b3ec892d5c0ce99ad82f2b0dc012ba3464c489bcf77b: Status 404 returned error can't find the container with id 7c7c1b113af7af1a50d4b3ec892d5c0ce99ad82f2b0dc012ba3464c489bcf77b Oct 31 03:39:36.082903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.082861 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" event={"ID":"be99c841-f0fd-4ede-a3c5-3a9c60f317c0","Type":"ContainerStarted","Data":"7c7c1b113af7af1a50d4b3ec892d5c0ce99ad82f2b0dc012ba3464c489bcf77b"} Oct 31 03:39:36.589060 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:39:36.589022 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:39:36.784857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.784822 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj"] Oct 31 03:39:36.791303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.791273 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:36.844137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.844024 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj"] Oct 31 03:39:36.929780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.929741 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw8ks\" (UniqueName: \"kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:36.929974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.929851 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:36.930065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:36.930051 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.031310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.031272 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.031500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.031432 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.031500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.031479 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8ks\" (UniqueName: \"kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.032141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.032075 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.034058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.034036 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.053261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.053226 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs"] Oct 31 03:39:37.056497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.056470 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62530746-r4kbs"] Oct 31 03:39:37.094812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.094717 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8ks\" (UniqueName: \"kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks\") pod \"maas-api-revoke-sub-4688bbf3-phszj\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.095609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.095580 4488 generic.go:356] "Generic (PLEG): container finished" podID="be99c841-f0fd-4ede-a3c5-3a9c60f317c0" containerID="96302356c75390148331f74ad2485a820515cb905ec0945ddd7416dab1632b77" exitCode=0 Oct 31 03:39:37.096967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.096942 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" event={"ID":"be99c841-f0fd-4ede-a3c5-3a9c60f317c0","Type":"ContainerDied","Data":"96302356c75390148331f74ad2485a820515cb905ec0945ddd7416dab1632b77"} Oct 31 03:39:37.106870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.106842 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:37.489839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:37.489808 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj"] Oct 31 03:39:37.490589 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:37.490567 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6546f9b9_badd_408e_ae87_4fc8299ba7e9.slice/crio-b06d1bfe64c066a0b1f206c1315ae4c6f3549885225edc2fe5e8a8dd32a0b132 WatchSource:0}: Error finding container b06d1bfe64c066a0b1f206c1315ae4c6f3549885225edc2fe5e8a8dd32a0b132: Status 404 returned error can't find the container with id b06d1bfe64c066a0b1f206c1315ae4c6f3549885225edc2fe5e8a8dd32a0b132 Oct 31 03:39:38.105763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.105665 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" event={"ID":"6546f9b9-badd-408e-ae87-4fc8299ba7e9","Type":"ContainerStarted","Data":"820cda6663b18efeb4136c3db812f5d8927b3c3340a74e11bb8e7f7c0979ce6d"} Oct 31 03:39:38.105763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.105732 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" event={"ID":"6546f9b9-badd-408e-ae87-4fc8299ba7e9","Type":"ContainerStarted","Data":"b06d1bfe64c066a0b1f206c1315ae4c6f3549885225edc2fe5e8a8dd32a0b132"} Oct 31 03:39:38.225619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.225562 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" podStartSLOduration=2.225547185 podStartE2EDuration="2.225547185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:38.221315133 +0000 UTC m=+1206.018033392" watchObservedRunningTime="2026-10-31 03:39:38.225547185 +0000 UTC m=+1206.022265442" Oct 31 03:39:38.262843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.262818 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:38.316701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.316654 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eceec002-bf44-414f-95e3-be7eb6c20d2b" path="/var/lib/kubelet/pods/eceec002-bf44-414f-95e3-be7eb6c20d2b/volumes" Oct 31 03:39:38.349351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.349316 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca\") pod \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " Oct 31 03:39:38.349525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.349384 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca\") pod \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " Oct 31 03:39:38.349525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.349488 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7\" (UniqueName: \"kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7\") pod \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\" (UID: \"be99c841-f0fd-4ede-a3c5-3a9c60f317c0\") " Oct 31 03:39:38.349799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.349770 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca" pod "be99c841-f0fd-4ede-a3c5-3a9c60f317c0" (UID: "be99c841-f0fd-4ede-a3c5-3a9c60f317c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:38.350061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.350042 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:38.351775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.351753 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca" pod "be99c841-f0fd-4ede-a3c5-3a9c60f317c0" (UID: "be99c841-f0fd-4ede-a3c5-3a9c60f317c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:38.351858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.351766 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7" pod "be99c841-f0fd-4ede-a3c5-3a9c60f317c0" (UID: "be99c841-f0fd-4ede-a3c5-3a9c60f317c0"). InnerVolumeSpecName "kube-api-access-cjvj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:38.451142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.451070 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:38.451142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.451138 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvj7\" (UniqueName: \"kubernetes.io/projected/be99c841-f0fd-4ede-a3c5-3a9c60f317c0-kube-api-access-cjvj7\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:38.869373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:38.869324 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:39.114576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:39.114540 4488 generic.go:356] "Generic (PLEG): container finished" podID="6546f9b9-badd-408e-ae87-4fc8299ba7e9" containerID="820cda6663b18efeb4136c3db812f5d8927b3c3340a74e11bb8e7f7c0979ce6d" exitCode=0 Oct 31 03:39:39.115033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:39.114617 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" event={"ID":"6546f9b9-badd-408e-ae87-4fc8299ba7e9","Type":"ContainerDied","Data":"820cda6663b18efeb4136c3db812f5d8927b3c3340a74e11bb8e7f7c0979ce6d"} Oct 31 03:39:39.116306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:39.116280 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" Oct 31 03:39:39.116455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:39.116303 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm" event={"ID":"be99c841-f0fd-4ede-a3c5-3a9c60f317c0","Type":"ContainerDied","Data":"7c7c1b113af7af1a50d4b3ec892d5c0ce99ad82f2b0dc012ba3464c489bcf77b"} Oct 31 03:39:39.116455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:39.116362 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7c1b113af7af1a50d4b3ec892d5c0ce99ad82f2b0dc012ba3464c489bcf77b" Oct 31 03:39:40.280024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.279999 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:40.377900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.377805 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks\" (UniqueName: \"kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks\") pod \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " Oct 31 03:39:40.377900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.377845 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca\") pod \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " Oct 31 03:39:40.378172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.377944 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca\") pod \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\" (UID: \"6546f9b9-badd-408e-ae87-4fc8299ba7e9\") " Oct 31 03:39:40.378395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.378363 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca" pod "6546f9b9-badd-408e-ae87-4fc8299ba7e9" (UID: "6546f9b9-badd-408e-ae87-4fc8299ba7e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:40.378618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.378591 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:40.380394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.380372 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks" pod "6546f9b9-badd-408e-ae87-4fc8299ba7e9" (UID: "6546f9b9-badd-408e-ae87-4fc8299ba7e9"). InnerVolumeSpecName "kube-api-access-fw8ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:40.380394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.380379 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca" pod "6546f9b9-badd-408e-ae87-4fc8299ba7e9" (UID: "6546f9b9-badd-408e-ae87-4fc8299ba7e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:40.480276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.480238 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw8ks\" (UniqueName: \"kubernetes.io/projected/6546f9b9-badd-408e-ae87-4fc8299ba7e9-kube-api-access-fw8ks\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:40.480276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:40.480271 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6546f9b9-badd-408e-ae87-4fc8299ba7e9-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:41.052167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.052130 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64"] Oct 31 03:39:41.058037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.058005 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795c5e7a-bjc64"] Oct 31 03:39:41.131900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.131865 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" event={"ID":"6546f9b9-badd-408e-ae87-4fc8299ba7e9","Type":"ContainerDied","Data":"b06d1bfe64c066a0b1f206c1315ae4c6f3549885225edc2fe5e8a8dd32a0b132"} Oct 31 03:39:41.131900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.131892 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj" Oct 31 03:39:41.132197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.131930 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b06d1bfe64c066a0b1f206c1315ae4c6f3549885225edc2fe5e8a8dd32a0b132" Oct 31 03:39:41.230552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.230518 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7"] Oct 31 03:39:41.231011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.230998 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6546f9b9-badd-408e-ae87-4fc8299ba7e9" containerName="revoke-keys" Oct 31 03:39:41.231053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.231013 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6546f9b9-badd-408e-ae87-4fc8299ba7e9" containerName="revoke-keys" Oct 31 03:39:41.231053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.231048 4488 cpu_manager.go:414] "containerMap: removing container" podUID="be99c841-f0fd-4ede-a3c5-3a9c60f317c0" containerName="revoke-keys" Oct 31 03:39:41.231143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.231053 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be99c841-f0fd-4ede-a3c5-3a9c60f317c0" containerName="revoke-keys" Oct 31 03:39:41.231173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.231152 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="be99c841-f0fd-4ede-a3c5-3a9c60f317c0" containerName="revoke-keys" Oct 31 03:39:41.231173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.231164 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6546f9b9-badd-408e-ae87-4fc8299ba7e9" containerName="revoke-keys" Oct 31 03:39:41.234851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.234819 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.297472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.297444 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7"] Oct 31 03:39:41.390488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390450 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.390488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390487 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.390700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390521 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.390700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390681 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.390772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390727 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.390839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390818 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7m78\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.390920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390906 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.391007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.390991 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.391041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.391035 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492011 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492079 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7m78\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492223 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492268 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492321 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492372 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492415 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492472 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492552 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492704 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492779 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492850 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.492899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.492866 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.493053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.493035 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.495124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.495062 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.495323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.495304 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.553382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.553351 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.553529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.553458 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7m78\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:41.850830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:41.850791 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:42.286995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:42.286958 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7"] Oct 31 03:39:42.287366 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:42.287342 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d365752_8a68_4d24_9b3d_ff4413476da4.slice/crio-b924ee0b8cb469a4e49ee62d4a5442e71f00c43fa1ff6398e69360fac8bf5e78 WatchSource:0}: Error finding container b924ee0b8cb469a4e49ee62d4a5442e71f00c43fa1ff6398e69360fac8bf5e78: Status 404 returned error can't find the container with id b924ee0b8cb469a4e49ee62d4a5442e71f00c43fa1ff6398e69360fac8bf5e78 Oct 31 03:39:42.289860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:42.289821 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:42.289950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:42.289894 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:42.289950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:42.289921 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:42.318543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:42.318515 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167d5711-a368-4260-a52d-07fc3f38b0e9" path="/var/lib/kubelet/pods/167d5711-a368-4260-a52d-07fc3f38b0e9/volumes" Oct 31 03:39:43.148293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:43.148249 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" event={"ID":"5d365752-8a68-4d24-9b3d-ff4413476da4","Type":"ContainerStarted","Data":"6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365"} Oct 31 03:39:43.148504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:43.148311 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" event={"ID":"5d365752-8a68-4d24-9b3d-ff4413476da4","Type":"ContainerStarted","Data":"b924ee0b8cb469a4e49ee62d4a5442e71f00c43fa1ff6398e69360fac8bf5e78"} Oct 31 03:39:43.700186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:43.700157 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:43.851739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:43.851699 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:43.853499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:43.853471 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:39:43.853628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:43.853522 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:39:44.851730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:44.851697 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:39:44.852204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:44.851756 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:39:45.851441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:45.851405 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:39:45.851650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:45.851479 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:39:46.085151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.085065 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podStartSLOduration=5.085049254 podStartE2EDuration="5.085049254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:43.328479195 +0000 UTC m=+1211.125197455" watchObservedRunningTime="2026-10-31 03:39:46.085049254 +0000 UTC m=+1213.881767513" Oct 31 03:39:46.087337 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.087316 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9"] Oct 31 03:39:46.091673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.091647 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.145177 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.145137 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:39:46.172158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.172116 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.172318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.172241 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.172318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.172292 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjksv\" (UniqueName: \"kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.194746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.194707 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:39:46.194746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.194737 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:39:46.194991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.194903 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:39:46.198679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.198655 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9"] Oct 31 03:39:46.273420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.273386 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.273610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.273460 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjksv\" (UniqueName: \"kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.273668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.273646 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.274153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.274078 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.276248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.276226 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.335491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.335457 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjksv\" (UniqueName: \"kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv\") pod \"maas-api-revoke-sub-eb037eba-2ndd9\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.430458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.430359 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:46.833529 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:46.833487 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fbda48_9568_431a_a8a1_86da1c8b791b.slice/crio-05ba49fba2a0cd7b8afb584e8d24787b631c8163ff1b10855edfa09ec7e9b376 WatchSource:0}: Error finding container 05ba49fba2a0cd7b8afb584e8d24787b631c8163ff1b10855edfa09ec7e9b376: Status 404 returned error can't find the container with id 05ba49fba2a0cd7b8afb584e8d24787b631c8163ff1b10855edfa09ec7e9b376 Oct 31 03:39:46.834509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.834480 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9"] Oct 31 03:39:46.851561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.851530 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:39:46.851658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.851593 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:39:46.899852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.899819 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb"] Oct 31 03:39:46.920450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.920416 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:46.962709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.962665 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb"] Oct 31 03:39:46.986380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.986344 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:46.986552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.986425 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:46.986627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:46.986537 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwzw\" (UniqueName: \"kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.088541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.088444 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.088541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.088509 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.089151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.088550 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwzw\" (UniqueName: \"kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.089408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.089377 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.091441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.091416 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.177435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.177395 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" event={"ID":"d7fbda48-9568-431a-a8a1-86da1c8b791b","Type":"ContainerStarted","Data":"05ba49fba2a0cd7b8afb584e8d24787b631c8163ff1b10855edfa09ec7e9b376"} Oct 31 03:39:47.226627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.225270 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwzw\" (UniqueName: \"kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw\") pod \"maas-api-revoke-sub-8b265bdb-h2mkb\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.243904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.243876 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:47.655127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.655078 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb"] Oct 31 03:39:47.656397 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:47.656371 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f514e46_ed61_49f9_93ac_19e73cb21544.slice/crio-10fc49f1bfb6006566b59e8755329d83b33c41dbb159e8c93e457022304027e4 WatchSource:0}: Error finding container 10fc49f1bfb6006566b59e8755329d83b33c41dbb159e8c93e457022304027e4: Status 404 returned error can't find the container with id 10fc49f1bfb6006566b59e8755329d83b33c41dbb159e8c93e457022304027e4 Oct 31 03:39:47.851602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.851567 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:39:47.851786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:47.851622 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:39:48.049269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.049242 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d"] Oct 31 03:39:48.052925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.052903 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ca67f3-dvd9d"] Oct 31 03:39:48.185623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.185584 4488 generic.go:356] "Generic (PLEG): container finished" podID="d7fbda48-9568-431a-a8a1-86da1c8b791b" containerID="9ca8c6f02d97999bf755196abfa961e87755e6ccd0141632ed78e91310d083f6" exitCode=0 Oct 31 03:39:48.186077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.185670 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" event={"ID":"d7fbda48-9568-431a-a8a1-86da1c8b791b","Type":"ContainerDied","Data":"9ca8c6f02d97999bf755196abfa961e87755e6ccd0141632ed78e91310d083f6"} Oct 31 03:39:48.187421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.187401 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" event={"ID":"5f514e46-ed61-49f9-93ac-19e73cb21544","Type":"ContainerStarted","Data":"9b4f59f76bd24850835a334fd365e5c1c6de5cea28b761bcc4485f24835ba6c5"} Oct 31 03:39:48.187528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.187437 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" event={"ID":"5f514e46-ed61-49f9-93ac-19e73cb21544","Type":"ContainerStarted","Data":"10fc49f1bfb6006566b59e8755329d83b33c41dbb159e8c93e457022304027e4"} Oct 31 03:39:48.318252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.318179 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670b7ee8-dedb-40f3-80f0-5c52ae241d8d" path="/var/lib/kubelet/pods/670b7ee8-dedb-40f3-80f0-5c52ae241d8d/volumes" Oct 31 03:39:48.424647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.424582 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" podStartSLOduration=2.424562595 podStartE2EDuration="2.424562595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:48.420215008 +0000 UTC m=+1216.216933268" watchObservedRunningTime="2026-10-31 03:39:48.424562595 +0000 UTC m=+1216.221280856" Oct 31 03:39:48.694306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.694275 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:48.851858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.851823 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:39:48.852072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:48.851882 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:39:49.199203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.199168 4488 generic.go:356] "Generic (PLEG): container finished" podID="5f514e46-ed61-49f9-93ac-19e73cb21544" containerID="9b4f59f76bd24850835a334fd365e5c1c6de5cea28b761bcc4485f24835ba6c5" exitCode=0 Oct 31 03:39:49.199674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.199259 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" event={"ID":"5f514e46-ed61-49f9-93ac-19e73cb21544","Type":"ContainerDied","Data":"9b4f59f76bd24850835a334fd365e5c1c6de5cea28b761bcc4485f24835ba6c5"} Oct 31 03:39:49.373774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.373747 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:49.418104 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.418049 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca\") pod \"d7fbda48-9568-431a-a8a1-86da1c8b791b\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " Oct 31 03:39:49.418280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.418232 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv\" (UniqueName: \"kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv\") pod \"d7fbda48-9568-431a-a8a1-86da1c8b791b\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " Oct 31 03:39:49.418366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.418299 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca\") pod \"d7fbda48-9568-431a-a8a1-86da1c8b791b\" (UID: \"d7fbda48-9568-431a-a8a1-86da1c8b791b\") " Oct 31 03:39:49.418466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.418441 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca" pod "d7fbda48-9568-431a-a8a1-86da1c8b791b" (UID: "d7fbda48-9568-431a-a8a1-86da1c8b791b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:49.418911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.418885 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:49.420648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.420618 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv" pod "d7fbda48-9568-431a-a8a1-86da1c8b791b" (UID: "d7fbda48-9568-431a-a8a1-86da1c8b791b"). InnerVolumeSpecName "kube-api-access-sjksv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:49.420648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.420618 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca" pod "d7fbda48-9568-431a-a8a1-86da1c8b791b" (UID: "d7fbda48-9568-431a-a8a1-86da1c8b791b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:49.519784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.519682 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjksv\" (UniqueName: \"kubernetes.io/projected/d7fbda48-9568-431a-a8a1-86da1c8b791b-kube-api-access-sjksv\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:49.519784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.519714 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7fbda48-9568-431a-a8a1-86da1c8b791b-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:49.855703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.855667 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:49.855987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.855950 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:49.856855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:49.856838 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:39:50.206955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.206875 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" Oct 31 03:39:50.206955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.206903 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9" event={"ID":"d7fbda48-9568-431a-a8a1-86da1c8b791b","Type":"ContainerDied","Data":"05ba49fba2a0cd7b8afb584e8d24787b631c8163ff1b10855edfa09ec7e9b376"} Oct 31 03:39:50.207414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.206973 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05ba49fba2a0cd7b8afb584e8d24787b631c8163ff1b10855edfa09ec7e9b376" Oct 31 03:39:50.365482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.365453 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:50.441810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.441776 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw\" (UniqueName: \"kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw\") pod \"5f514e46-ed61-49f9-93ac-19e73cb21544\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " Oct 31 03:39:50.442001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.441860 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca\") pod \"5f514e46-ed61-49f9-93ac-19e73cb21544\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " Oct 31 03:39:50.442123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.442057 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca\") pod \"5f514e46-ed61-49f9-93ac-19e73cb21544\" (UID: \"5f514e46-ed61-49f9-93ac-19e73cb21544\") " Oct 31 03:39:50.442250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.442225 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca" pod "5f514e46-ed61-49f9-93ac-19e73cb21544" (UID: "5f514e46-ed61-49f9-93ac-19e73cb21544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:50.442503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.442489 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:50.444268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.444241 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca" pod "5f514e46-ed61-49f9-93ac-19e73cb21544" (UID: "5f514e46-ed61-49f9-93ac-19e73cb21544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:50.444357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.444273 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw" pod "5f514e46-ed61-49f9-93ac-19e73cb21544" (UID: "5f514e46-ed61-49f9-93ac-19e73cb21544"). InnerVolumeSpecName "kube-api-access-bfwzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:50.543488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.543397 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f514e46-ed61-49f9-93ac-19e73cb21544-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:50.543488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:50.543426 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwzw\" (UniqueName: \"kubernetes.io/projected/5f514e46-ed61-49f9-93ac-19e73cb21544-kube-api-access-bfwzw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:51.216570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:51.216540 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" Oct 31 03:39:51.217040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:51.216568 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb" event={"ID":"5f514e46-ed61-49f9-93ac-19e73cb21544","Type":"ContainerDied","Data":"10fc49f1bfb6006566b59e8755329d83b33c41dbb159e8c93e457022304027e4"} Oct 31 03:39:51.217040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:51.216627 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10fc49f1bfb6006566b59e8755329d83b33c41dbb159e8c93e457022304027e4" Oct 31 03:39:52.318562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.318534 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b"] Oct 31 03:39:52.319050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.318977 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d7fbda48-9568-431a-a8a1-86da1c8b791b" containerName="revoke-keys" Oct 31 03:39:52.319050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.318989 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fbda48-9568-431a-a8a1-86da1c8b791b" containerName="revoke-keys" Oct 31 03:39:52.319050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.318999 4488 cpu_manager.go:414] "containerMap: removing container" podUID="5f514e46-ed61-49f9-93ac-19e73cb21544" containerName="revoke-keys" Oct 31 03:39:52.319050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.319004 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f514e46-ed61-49f9-93ac-19e73cb21544" containerName="revoke-keys" Oct 31 03:39:52.319232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.319132 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fbda48-9568-431a-a8a1-86da1c8b791b" containerName="revoke-keys" Oct 31 03:39:52.319232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.319141 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f514e46-ed61-49f9-93ac-19e73cb21544" containerName="revoke-keys" Oct 31 03:39:52.322157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.322141 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.364536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.364503 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.364719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.364564 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5xn\" (UniqueName: \"kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.364719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.364680 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.425521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.425476 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:39:52.425521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.425479 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:39:52.425521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.425481 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:39:52.425821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.425538 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:39:52.429484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.429458 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b"] Oct 31 03:39:52.465514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.465467 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.465701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.465538 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5xn\" (UniqueName: \"kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.465701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.465601 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.466308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.466286 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.468305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.468283 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.527681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.527640 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5xn\" (UniqueName: \"kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn\") pod \"maas-api-revoke-sub-7edba4ad-nsg8b\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:52.635862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:52.635823 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:53.016172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:53.016143 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b"] Oct 31 03:39:53.017322 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:53.017295 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ae3bea_90ba_4e93_b928_9ab3fb43b855.slice/crio-c3310fc714e3dcf699e8a12ac797d4805abf2c944f490cfbc477204b04c261c2 WatchSource:0}: Error finding container c3310fc714e3dcf699e8a12ac797d4805abf2c944f490cfbc477204b04c261c2: Status 404 returned error can't find the container with id c3310fc714e3dcf699e8a12ac797d4805abf2c944f490cfbc477204b04c261c2 Oct 31 03:39:53.232017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:53.231982 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" event={"ID":"13ae3bea-90ba-4e93-b928-9ab3fb43b855","Type":"ContainerStarted","Data":"c3310fc714e3dcf699e8a12ac797d4805abf2c944f490cfbc477204b04c261c2"} Oct 31 03:39:53.515131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:53.515081 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:54.240881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:54.240844 4488 generic.go:356] "Generic (PLEG): container finished" podID="13ae3bea-90ba-4e93-b928-9ab3fb43b855" containerID="5e1bfdb39386fbb7d88165b6374549a33030e5ac6b69d8320c094124b90e625e" exitCode=0 Oct 31 03:39:54.241073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:54.240920 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" event={"ID":"13ae3bea-90ba-4e93-b928-9ab3fb43b855","Type":"ContainerDied","Data":"5e1bfdb39386fbb7d88165b6374549a33030e5ac6b69d8320c094124b90e625e"} Oct 31 03:39:55.088301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.088264 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m"] Oct 31 03:39:55.092648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.092619 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.199050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.199013 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m"] Oct 31 03:39:55.199623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.199597 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.199725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.199649 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvmq\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.199814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.199787 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.199901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.199880 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.200012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.199994 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.200076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.200061 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.200188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.200172 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.200252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.200239 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.200327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.200310 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.270238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.270205 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.301504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301473 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301542 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301597 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301635 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301665 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301707 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301762 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvmq\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301809 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.301902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301865 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.302128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.301922 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.302128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.302021 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.302233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.302207 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.302303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.302283 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.302363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.302316 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.304284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.304261 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.304503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.304482 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.363613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.363535 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.363613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.363581 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvmq\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.403283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.403248 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:55.410859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.410829 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:55.426995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.426960 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.504933 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.504892 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn\" (UniqueName: \"kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn\") pod \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " Oct 31 03:39:55.505164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.505015 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca\") pod \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " Oct 31 03:39:55.505236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.505204 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca\") pod \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\" (UID: \"13ae3bea-90ba-4e93-b928-9ab3fb43b855\") " Oct 31 03:39:55.505572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.505546 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca" pod "13ae3bea-90ba-4e93-b928-9ab3fb43b855" (UID: "13ae3bea-90ba-4e93-b928-9ab3fb43b855"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:55.505702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.505687 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:55.508404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.508179 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca" pod "13ae3bea-90ba-4e93-b928-9ab3fb43b855" (UID: "13ae3bea-90ba-4e93-b928-9ab3fb43b855"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:55.509117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.508927 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn" pod "13ae3bea-90ba-4e93-b928-9ab3fb43b855" (UID: "13ae3bea-90ba-4e93-b928-9ab3fb43b855"). InnerVolumeSpecName "kube-api-access-qc5xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:55.607044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.607010 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc5xn\" (UniqueName: \"kubernetes.io/projected/13ae3bea-90ba-4e93-b928-9ab3fb43b855-kube-api-access-qc5xn\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:55.607044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.607042 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae3bea-90ba-4e93-b928-9ab3fb43b855-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:39:55.847075 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.847045 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m"] Oct 31 03:39:55.847692 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:39:55.847652 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee9ef6f_23a4_40bc_af93_4d4a41a8bc21.slice/crio-18f40c2727e11767f5c64b4929ffb2fd8aa35643ff3187d48f344b7db4c33f70 WatchSource:0}: Error finding container 18f40c2727e11767f5c64b4929ffb2fd8aa35643ff3187d48f344b7db4c33f70: Status 404 returned error can't find the container with id 18f40c2727e11767f5c64b4929ffb2fd8aa35643ff3187d48f344b7db4c33f70 Oct 31 03:39:55.850455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.850422 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:55.850531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.850492 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:55.850531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:55.850520 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:39:56.258764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.258657 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" event={"ID":"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21","Type":"ContainerStarted","Data":"a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963"} Oct 31 03:39:56.258764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.258713 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" event={"ID":"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21","Type":"ContainerStarted","Data":"18f40c2727e11767f5c64b4929ffb2fd8aa35643ff3187d48f344b7db4c33f70"} Oct 31 03:39:56.260299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.260275 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" Oct 31 03:39:56.260446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.260303 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b" event={"ID":"13ae3bea-90ba-4e93-b928-9ab3fb43b855","Type":"ContainerDied","Data":"c3310fc714e3dcf699e8a12ac797d4805abf2c944f490cfbc477204b04c261c2"} Oct 31 03:39:56.260446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.260342 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3310fc714e3dcf699e8a12ac797d4805abf2c944f490cfbc477204b04c261c2" Oct 31 03:39:56.382629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.382579 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podStartSLOduration=1.382564117 podStartE2EDuration="1.382564117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:56.380634778 +0000 UTC m=+1224.177353064" watchObservedRunningTime="2026-10-31 03:39:56.382564117 +0000 UTC m=+1224.179282373" Oct 31 03:39:56.411050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.411015 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:39:56.412521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.412495 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:39:56.412632 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:56.412555 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:39:57.430002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:57.429960 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:39:57.430641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:57.430593 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:39:58.401242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:58.401187 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:58.424636 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:58.424600 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:39:58.424955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:58.424916 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:39:58.872645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:58.872608 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:39:59.412707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:59.412671 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:39:59.413017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:39:59.412989 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:40:00.411757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:00.411722 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:40:00.412833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:00.412399 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:40:01.073262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.073207 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.087232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.087185 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.091229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.091173 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.091839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.091808 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.092316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.092291 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:40:01.104507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.104462 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:40:01.105597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.105568 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.110880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.110169 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.168659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.168603 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.326175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.322924 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.443116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.425054 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:40:01.452242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.452191 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:40:01.485345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.485304 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.606871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.606813 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.734704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.734655 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.799322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.799261 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:01.914401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:01.914298 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.054124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.053378 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.075242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.075192 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.075987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.075966 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:40:02.088779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.088740 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.091559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.091531 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:40:02.102120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.092653 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.102802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.102759 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.103329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.103298 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:40:02.103739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.103722 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:40:02.103901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.103878 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:40:02.104045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.104035 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:40:02.115250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.115191 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.115911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.115863 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:40:02.116449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.116428 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:40:02.116614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.116600 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:40:02.142225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.142174 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.143325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.143299 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:40:02.214049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.213947 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 03:40:02.235266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.235212 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.236007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.235978 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:40:02.274221 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.274162 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:02.321131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.297538 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.380115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.354702 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:02.411460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.411420 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:40:02.411775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.411741 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:40:02.459033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.458998 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:40:02.498351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.498268 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:40:02.569217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.569187 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:40:02.601187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.575018 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:40:02.608240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.608196 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:40:02.609457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.609427 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:40:02.648534 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.648504 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:40:02.681805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.681777 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:40:02.690792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.690765 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:40:02.692942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.692921 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:40:02.740023 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.739989 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:40:02.745755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.744136 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:40:02.745755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.744197 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:40:02.751778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:02.751690 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:40:03.412553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.412512 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:40:03.412733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.412586 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:40:03.436949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.436917 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:03.539481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.539446 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn"] Oct 31 03:40:03.540137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.540112 4488 cpu_manager.go:414] "containerMap: removing container" podUID="13ae3bea-90ba-4e93-b928-9ab3fb43b855" containerName="revoke-keys" Oct 31 03:40:03.540308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.540279 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ae3bea-90ba-4e93-b928-9ab3fb43b855" containerName="revoke-keys" Oct 31 03:40:03.540499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.540484 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ae3bea-90ba-4e93-b928-9ab3fb43b855" containerName="revoke-keys" Oct 31 03:40:03.545630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.545604 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.600663 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.600631 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:40:03.600890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.600749 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:40:03.600890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.600806 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:40:03.600890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.600846 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:40:03.604067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.604038 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn"] Oct 31 03:40:03.701941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.701836 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zgg\" (UniqueName: \"kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.702147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.701948 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.702147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.702038 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.803481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.803446 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.803678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.803537 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.803747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.803720 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zgg\" (UniqueName: \"kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.804289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.804261 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.808572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.808541 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:03.865836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:03.865802 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zgg\" (UniqueName: \"kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg\") pod \"maas-api-revoke-sub-5f74683d-qpxsn\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:04.165231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:04.165195 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:04.416656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:04.416577 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:40:04.416896 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:04.416877 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:40:04.417649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:04.417631 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:40:04.554782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:04.554751 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn"] Oct 31 03:40:04.555907 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:04.555870 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8708546d_ea7d_470c_a6af_1a17a5376977.slice/crio-8cef464a4028a7e4027381ac6bf7b9a2750237698eabe8d38b320b8cc5d597a9 WatchSource:0}: Error finding container 8cef464a4028a7e4027381ac6bf7b9a2750237698eabe8d38b320b8cc5d597a9: Status 404 returned error can't find the container with id 8cef464a4028a7e4027381ac6bf7b9a2750237698eabe8d38b320b8cc5d597a9 Oct 31 03:40:05.521913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:05.521876 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" event={"ID":"8708546d-ea7d-470c-a6af-1a17a5376977","Type":"ContainerStarted","Data":"d0f6db342c89f65e6489c6f9dadc57bb1401041ed51f9d48deb6c279ef820bfa"} Oct 31 03:40:05.522160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:05.521934 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" event={"ID":"8708546d-ea7d-470c-a6af-1a17a5376977","Type":"ContainerStarted","Data":"8cef464a4028a7e4027381ac6bf7b9a2750237698eabe8d38b320b8cc5d597a9"} Oct 31 03:40:05.657362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:05.657300 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" podStartSLOduration=2.6572863829999998 podStartE2EDuration="2.657286383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:05.655185509 +0000 UTC m=+1233.451903771" watchObservedRunningTime="2026-10-31 03:40:05.657286383 +0000 UTC m=+1233.454004641" Oct 31 03:40:06.530448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:06.530414 4488 generic.go:356] "Generic (PLEG): container finished" podID="8708546d-ea7d-470c-a6af-1a17a5376977" containerID="d0f6db342c89f65e6489c6f9dadc57bb1401041ed51f9d48deb6c279ef820bfa" exitCode=0 Oct 31 03:40:06.530663 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:06.530502 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" event={"ID":"8708546d-ea7d-470c-a6af-1a17a5376977","Type":"ContainerDied","Data":"d0f6db342c89f65e6489c6f9dadc57bb1401041ed51f9d48deb6c279ef820bfa"} Oct 31 03:40:06.589906 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:40:06.589867 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:40:07.704718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.704693 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:07.753859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.753828 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg\" (UniqueName: \"kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg\") pod \"8708546d-ea7d-470c-a6af-1a17a5376977\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " Oct 31 03:40:07.754050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.753865 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca\") pod \"8708546d-ea7d-470c-a6af-1a17a5376977\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " Oct 31 03:40:07.754050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.753937 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca\") pod \"8708546d-ea7d-470c-a6af-1a17a5376977\" (UID: \"8708546d-ea7d-470c-a6af-1a17a5376977\") " Oct 31 03:40:07.754615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.754574 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca" pod "8708546d-ea7d-470c-a6af-1a17a5376977" (UID: "8708546d-ea7d-470c-a6af-1a17a5376977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:07.756486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.756420 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca" pod "8708546d-ea7d-470c-a6af-1a17a5376977" (UID: "8708546d-ea7d-470c-a6af-1a17a5376977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:07.756766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.756744 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg" pod "8708546d-ea7d-470c-a6af-1a17a5376977" (UID: "8708546d-ea7d-470c-a6af-1a17a5376977"). InnerVolumeSpecName "kube-api-access-p2zgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:07.856326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.856290 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2zgg\" (UniqueName: \"kubernetes.io/projected/8708546d-ea7d-470c-a6af-1a17a5376977-kube-api-access-p2zgg\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:07.856326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.856320 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:07.856326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:07.856329 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8708546d-ea7d-470c-a6af-1a17a5376977-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:08.212012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:08.211857 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:08.549427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:08.549343 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" Oct 31 03:40:08.549427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:08.549371 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn" event={"ID":"8708546d-ea7d-470c-a6af-1a17a5376977","Type":"ContainerDied","Data":"8cef464a4028a7e4027381ac6bf7b9a2750237698eabe8d38b320b8cc5d597a9"} Oct 31 03:40:08.549427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:08.549419 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cef464a4028a7e4027381ac6bf7b9a2750237698eabe8d38b320b8cc5d597a9" Oct 31 03:40:10.049323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.049286 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9"] Oct 31 03:40:10.099292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.099260 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4977a29-lbdn9"] Oct 31 03:40:10.317347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.317268 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b088be40-10c1-43c9-a94e-e60a05190ee7" path="/var/lib/kubelet/pods/b088be40-10c1-43c9-a94e-e60a05190ee7/volumes" Oct 31 03:40:10.929295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.929257 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn"] Oct 31 03:40:10.929812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.929798 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8708546d-ea7d-470c-a6af-1a17a5376977" containerName="revoke-keys" Oct 31 03:40:10.929858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.929814 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8708546d-ea7d-470c-a6af-1a17a5376977" containerName="revoke-keys" Oct 31 03:40:10.929924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.929916 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8708546d-ea7d-470c-a6af-1a17a5376977" containerName="revoke-keys" Oct 31 03:40:10.934527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.934509 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:10.999844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.999809 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.999923 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmkv\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:10.999992 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.000015 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.000077 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.000180 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.000214 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.000247 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.000340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.000304 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.041954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.041923 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn"] Oct 31 03:40:11.102204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102167 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102234 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmkv\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102275 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102297 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102342 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102403 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102437 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102472 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102493 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.102677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102615 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.103172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102713 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.103172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102841 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.103172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.102942 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.103487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.103332 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.104937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.104913 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.105204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.105186 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.163850 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.163815 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.164032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.163893 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmkv\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.249429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.249339 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:11.688158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.688119 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn"] Oct 31 03:40:11.688889 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:11.688860 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64704aa_9fdd_4ac0_843c_7f699375d126.slice/crio-0c15b97c4d51e2114643ec7e4f5d45f05f4b290b70b5d7d2dbe2ea0791500428 WatchSource:0}: Error finding container 0c15b97c4d51e2114643ec7e4f5d45f05f4b290b70b5d7d2dbe2ea0791500428: Status 404 returned error can't find the container with id 0c15b97c4d51e2114643ec7e4f5d45f05f4b290b70b5d7d2dbe2ea0791500428 Oct 31 03:40:11.691983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.691943 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:11.692160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.692039 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:11.692160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.692102 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:11.956856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.956772 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd"] Oct 31 03:40:11.960823 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:11.960803 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.014450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.014398 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:40:12.014650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.014584 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:40:12.014709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.014644 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:40:12.014863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.014825 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:40:12.015492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.015341 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.015492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.015425 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzdv\" (UniqueName: \"kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.015692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.015631 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.020921 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.020895 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd"] Oct 31 03:40:12.051444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.051410 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx"] Oct 31 03:40:12.057451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.057421 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-643713c8-jjknx"] Oct 31 03:40:12.117542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.117511 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.118011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.117572 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzdv\" (UniqueName: \"kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.118011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.117670 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.118392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.118374 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.120017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.119997 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.178937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.178904 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzdv\" (UniqueName: \"kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv\") pod \"maas-api-revoke-sub-29fa3641-5zdwd\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.276914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.276815 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:12.343806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.343759 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02603570-fcca-4fe6-9c0a-b67c402bdd1b" path="/var/lib/kubelet/pods/02603570-fcca-4fe6-9c0a-b67c402bdd1b/volumes" Oct 31 03:40:12.580656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.580612 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" event={"ID":"b64704aa-9fdd-4ac0-843c-7f699375d126","Type":"ContainerStarted","Data":"403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0"} Oct 31 03:40:12.580854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.580685 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" event={"ID":"b64704aa-9fdd-4ac0-843c-7f699375d126","Type":"ContainerStarted","Data":"0c15b97c4d51e2114643ec7e4f5d45f05f4b290b70b5d7d2dbe2ea0791500428"} Oct 31 03:40:12.688976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.688946 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd"] Oct 31 03:40:12.689624 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:12.689593 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a119fa_5656_410b_94eb_f8becd5a1df6.slice/crio-9bc01ab08838e94f117bd80e82bd3a0e21e159433fd4b85afe0fed3b2bd940f2 WatchSource:0}: Error finding container 9bc01ab08838e94f117bd80e82bd3a0e21e159433fd4b85afe0fed3b2bd940f2: Status 404 returned error can't find the container with id 9bc01ab08838e94f117bd80e82bd3a0e21e159433fd4b85afe0fed3b2bd940f2 Oct 31 03:40:12.703869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:12.703810 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podStartSLOduration=2.703793737 podStartE2EDuration="2.703793737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:12.70220681 +0000 UTC m=+1240.498925073" watchObservedRunningTime="2026-10-31 03:40:12.703793737 +0000 UTC m=+1240.500512000" Oct 31 03:40:13.134147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.131866 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:13.250541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.250444 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:13.252014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.251989 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:40:13.252157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.252051 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:40:13.589465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.589429 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" event={"ID":"c5a119fa-5656-410b-94eb-f8becd5a1df6","Type":"ContainerStarted","Data":"045335e4c30657ffe7549213646ea1ee7f5570cb70aa20501ad07707a9853192"} Oct 31 03:40:13.589646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.589490 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" event={"ID":"c5a119fa-5656-410b-94eb-f8becd5a1df6","Type":"ContainerStarted","Data":"9bc01ab08838e94f117bd80e82bd3a0e21e159433fd4b85afe0fed3b2bd940f2"} Oct 31 03:40:13.735264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:13.735207 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" podStartSLOduration=2.735190865 podStartE2EDuration="2.735190865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:13.708569732 +0000 UTC m=+1241.505287991" watchObservedRunningTime="2026-10-31 03:40:13.735190865 +0000 UTC m=+1241.531909122" Oct 31 03:40:14.250410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.250379 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:40:14.250780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.250448 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:40:14.598318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.598288 4488 generic.go:356] "Generic (PLEG): container finished" podID="c5a119fa-5656-410b-94eb-f8becd5a1df6" containerID="045335e4c30657ffe7549213646ea1ee7f5570cb70aa20501ad07707a9853192" exitCode=0 Oct 31 03:40:14.598483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.598369 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" event={"ID":"c5a119fa-5656-410b-94eb-f8becd5a1df6","Type":"ContainerDied","Data":"045335e4c30657ffe7549213646ea1ee7f5570cb70aa20501ad07707a9853192"} Oct 31 03:40:14.843115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.841513 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz"] Oct 31 03:40:14.846597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.846576 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:14.900133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.900061 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz"] Oct 31 03:40:14.968225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.968179 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:14.968436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.968356 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxp9\" (UniqueName: \"kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:14.968602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:14.968577 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.070291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.070254 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxp9\" (UniqueName: \"kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.070458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.070415 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.070526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.070492 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.071280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.071184 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.072992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.072974 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.137483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.137408 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxp9\" (UniqueName: \"kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9\") pod \"maas-api-revoke-sub-5429b4a8-zhqzz\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.161403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.161374 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:15.249651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.249619 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:40:15.249767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.249676 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:40:15.576973 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:15.576916 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdab6af_cac4_4bbb_9395_a1f2dee9a5ca.slice/crio-26139b03b1a0a0f37e44ce1db2110a55f0fd3b4cb60ebb19913a0a1bf06f8b95 WatchSource:0}: Error finding container 26139b03b1a0a0f37e44ce1db2110a55f0fd3b4cb60ebb19913a0a1bf06f8b95: Status 404 returned error can't find the container with id 26139b03b1a0a0f37e44ce1db2110a55f0fd3b4cb60ebb19913a0a1bf06f8b95 Oct 31 03:40:15.577474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.577295 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz"] Oct 31 03:40:15.639115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.637141 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" event={"ID":"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca","Type":"ContainerStarted","Data":"26139b03b1a0a0f37e44ce1db2110a55f0fd3b4cb60ebb19913a0a1bf06f8b95"} Oct 31 03:40:15.841055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.841024 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:15.981574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.981539 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv\" (UniqueName: \"kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv\") pod \"c5a119fa-5656-410b-94eb-f8becd5a1df6\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " Oct 31 03:40:15.981773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.981646 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca\") pod \"c5a119fa-5656-410b-94eb-f8becd5a1df6\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " Oct 31 03:40:15.981955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.981918 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca\") pod \"c5a119fa-5656-410b-94eb-f8becd5a1df6\" (UID: \"c5a119fa-5656-410b-94eb-f8becd5a1df6\") " Oct 31 03:40:15.982127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.982036 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca" pod "c5a119fa-5656-410b-94eb-f8becd5a1df6" (UID: "c5a119fa-5656-410b-94eb-f8becd5a1df6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:15.982430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.982412 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:15.984077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.984051 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv" pod "c5a119fa-5656-410b-94eb-f8becd5a1df6" (UID: "c5a119fa-5656-410b-94eb-f8becd5a1df6"). InnerVolumeSpecName "kube-api-access-5pzdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:15.984348 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:15.984332 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca" pod "c5a119fa-5656-410b-94eb-f8becd5a1df6" (UID: "c5a119fa-5656-410b-94eb-f8becd5a1df6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:16.083727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.083688 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a119fa-5656-410b-94eb-f8becd5a1df6-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:16.083868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.083730 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzdv\" (UniqueName: \"kubernetes.io/projected/c5a119fa-5656-410b-94eb-f8becd5a1df6-kube-api-access-5pzdv\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:16.250746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.250664 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:40:16.250746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.250720 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:40:16.645032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.645000 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" Oct 31 03:40:16.645032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.645008 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd" event={"ID":"c5a119fa-5656-410b-94eb-f8becd5a1df6","Type":"ContainerDied","Data":"9bc01ab08838e94f117bd80e82bd3a0e21e159433fd4b85afe0fed3b2bd940f2"} Oct 31 03:40:16.645569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.645062 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc01ab08838e94f117bd80e82bd3a0e21e159433fd4b85afe0fed3b2bd940f2" Oct 31 03:40:16.646838 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.646814 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" event={"ID":"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca","Type":"ContainerStarted","Data":"ed869ac7baaa7c974728c827b04871d33eb723218f65a09c23937be5108b04c9"} Oct 31 03:40:16.830904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:16.830839 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" podStartSLOduration=2.830819505 podStartE2EDuration="2.830819505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:16.803327212 +0000 UTC m=+1244.600045471" watchObservedRunningTime="2026-10-31 03:40:16.830819505 +0000 UTC m=+1244.627537762" Oct 31 03:40:17.249957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:17.249915 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:40:17.250218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:17.249989 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:40:17.662005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:17.661970 4488 generic.go:356] "Generic (PLEG): container finished" podID="2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" containerID="ed869ac7baaa7c974728c827b04871d33eb723218f65a09c23937be5108b04c9" exitCode=0 Oct 31 03:40:17.662545 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:17.662068 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" event={"ID":"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca","Type":"ContainerDied","Data":"ed869ac7baaa7c974728c827b04871d33eb723218f65a09c23937be5108b04c9"} Oct 31 03:40:18.010744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.010651 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:18.250225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.250180 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:40:18.250414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.250254 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:40:18.900036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.899998 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt"] Oct 31 03:40:18.901014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.900989 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:18.901014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.901005 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c5a119fa-5656-410b-94eb-f8becd5a1df6" containerName="revoke-keys" Oct 31 03:40:18.901276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.901026 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a119fa-5656-410b-94eb-f8becd5a1df6" containerName="revoke-keys" Oct 31 03:40:18.901335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.901280 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" containerName="revoke-keys" Oct 31 03:40:18.901335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.901301 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a119fa-5656-410b-94eb-f8becd5a1df6" containerName="revoke-keys" Oct 31 03:40:18.905422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.905394 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:18.960640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:18.960553 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt"] Oct 31 03:40:19.024001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.023962 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca\") pod \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " Oct 31 03:40:19.024205 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.024131 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca\") pod \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " Oct 31 03:40:19.024321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.024303 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9\" (UniqueName: \"kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9\") pod \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\" (UID: \"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca\") " Oct 31 03:40:19.024564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.024534 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca" pod "2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" (UID: "2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:19.024678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.024583 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58nfm\" (UniqueName: \"kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.024789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.024773 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.024934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.024916 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.025141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.025071 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:19.026418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.026390 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9" pod "2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" (UID: "2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca"). InnerVolumeSpecName "kube-api-access-cqxp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:19.026418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.026398 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca" pod "2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" (UID: "2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:19.126209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.126173 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.126387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.126265 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.126387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.126318 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58nfm\" (UniqueName: \"kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.126387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.126382 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:19.126477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.126393 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxp9\" (UniqueName: \"kubernetes.io/projected/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca-kube-api-access-cqxp9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:19.126981 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.126900 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.128702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.128677 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.188308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.188272 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58nfm\" (UniqueName: \"kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm\") pod \"maas-api-revoke-sub-efaa3bc7-gc6bt\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.222550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.222464 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:19.255431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.255404 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:19.255867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.255847 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:19.256833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.256811 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:40:19.606905 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.606875 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt"] Oct 31 03:40:19.607536 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:19.607511 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0942e8_57e0_49b4_8d8f_1dfe6354f8d5.slice/crio-6b9f6c2da823ded842c8757756e1cbf529d2b644178d5e3dd19beb7d48725611 WatchSource:0}: Error finding container 6b9f6c2da823ded842c8757756e1cbf529d2b644178d5e3dd19beb7d48725611: Status 404 returned error can't find the container with id 6b9f6c2da823ded842c8757756e1cbf529d2b644178d5e3dd19beb7d48725611 Oct 31 03:40:19.687453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.687414 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" event={"ID":"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5","Type":"ContainerStarted","Data":"6b9f6c2da823ded842c8757756e1cbf529d2b644178d5e3dd19beb7d48725611"} Oct 31 03:40:19.689284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.689259 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" Oct 31 03:40:19.689406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.689279 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz" event={"ID":"2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca","Type":"ContainerDied","Data":"26139b03b1a0a0f37e44ce1db2110a55f0fd3b4cb60ebb19913a0a1bf06f8b95"} Oct 31 03:40:19.689406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:19.689325 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26139b03b1a0a0f37e44ce1db2110a55f0fd3b4cb60ebb19913a0a1bf06f8b95" Oct 31 03:40:20.049823 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.049788 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6"] Oct 31 03:40:20.053625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.053592 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5806d66f-nlhm6"] Oct 31 03:40:20.318376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.318299 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a55e462-2fcf-4c1c-b14e-395308653cb2" path="/var/lib/kubelet/pods/9a55e462-2fcf-4c1c-b14e-395308653cb2/volumes" Oct 31 03:40:20.320261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.320238 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj"] Oct 31 03:40:20.320726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.320714 4488 cpu_manager.go:414] "containerMap: removing container" podUID="2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" containerName="revoke-keys" Oct 31 03:40:20.320767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.320727 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" containerName="revoke-keys" Oct 31 03:40:20.326008 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.325993 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.384943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.384911 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj"] Oct 31 03:40:20.445181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.445143 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.445433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.445413 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvtm\" (UniqueName: \"kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.445503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.445447 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.547241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.547211 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvtm\" (UniqueName: \"kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.547241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.547246 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.547420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.547310 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.547930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.547900 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.549957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.549937 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.610686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.610655 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvtm\" (UniqueName: \"kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm\") pod \"maas-api-revoke-sub-81898ccb-2lxcj\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.639598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.639561 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:20.701295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.700942 4488 generic.go:356] "Generic (PLEG): container finished" podID="bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" containerID="2a2bfa55ba7b861aeac331f654e32ec446fee0a9976643a91467844e3f128c35" exitCode=0 Oct 31 03:40:20.701295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:20.701160 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" event={"ID":"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5","Type":"ContainerDied","Data":"2a2bfa55ba7b861aeac331f654e32ec446fee0a9976643a91467844e3f128c35"} Oct 31 03:40:21.032138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:21.032105 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj"] Oct 31 03:40:21.032321 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:21.032298 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5605ed00_fdcc_4d12_8355_1fbc8e4ba903.slice/crio-214427464dadc5c02f75c07568bc0ba1d0a0b033654814e9fa33f958e5e25c19 WatchSource:0}: Error finding container 214427464dadc5c02f75c07568bc0ba1d0a0b033654814e9fa33f958e5e25c19: Status 404 returned error can't find the container with id 214427464dadc5c02f75c07568bc0ba1d0a0b033654814e9fa33f958e5e25c19 Oct 31 03:40:21.713040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:21.713002 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" event={"ID":"5605ed00-fdcc-4d12-8355-1fbc8e4ba903","Type":"ContainerStarted","Data":"5f1d38281c6beff6fc3a808c7beb75131250b3738accb20c233f44db5c1573d2"} Oct 31 03:40:21.713541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:21.713071 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" event={"ID":"5605ed00-fdcc-4d12-8355-1fbc8e4ba903","Type":"ContainerStarted","Data":"214427464dadc5c02f75c07568bc0ba1d0a0b033654814e9fa33f958e5e25c19"} Oct 31 03:40:21.832844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:21.832781 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" podStartSLOduration=1.832760908 podStartE2EDuration="1.832760908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:21.83087076 +0000 UTC m=+1249.627589018" watchObservedRunningTime="2026-10-31 03:40:21.832760908 +0000 UTC m=+1249.629479197" Oct 31 03:40:21.906329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:21.906300 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:22.075036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.074987 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca\") pod \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " Oct 31 03:40:22.075284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.075069 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm\" (UniqueName: \"kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm\") pod \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " Oct 31 03:40:22.075284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.075166 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca\") pod \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\" (UID: \"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5\") " Oct 31 03:40:22.075542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.075508 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca" pod "bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" (UID: "bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:22.075772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.075755 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:22.077593 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.077563 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm" pod "bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" (UID: "bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5"). InnerVolumeSpecName "kube-api-access-58nfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:22.077797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.077772 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca" pod "bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" (UID: "bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:22.177641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.177606 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:22.177641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.177637 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58nfm\" (UniqueName: \"kubernetes.io/projected/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5-kube-api-access-58nfm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:22.721990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.721962 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" Oct 31 03:40:22.722465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.721963 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt" event={"ID":"bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5","Type":"ContainerDied","Data":"6b9f6c2da823ded842c8757756e1cbf529d2b644178d5e3dd19beb7d48725611"} Oct 31 03:40:22.722465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.722125 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b9f6c2da823ded842c8757756e1cbf529d2b644178d5e3dd19beb7d48725611" Oct 31 03:40:22.723775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.723750 4488 generic.go:356] "Generic (PLEG): container finished" podID="5605ed00-fdcc-4d12-8355-1fbc8e4ba903" containerID="5f1d38281c6beff6fc3a808c7beb75131250b3738accb20c233f44db5c1573d2" exitCode=0 Oct 31 03:40:22.723894 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.723857 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" event={"ID":"5605ed00-fdcc-4d12-8355-1fbc8e4ba903","Type":"ContainerDied","Data":"5f1d38281c6beff6fc3a808c7beb75131250b3738accb20c233f44db5c1573d2"} Oct 31 03:40:22.825885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:22.825844 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:23.314147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:23.314114 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:23.897464 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:23.897436 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:24.002166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.002033 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca\") pod \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " Oct 31 03:40:24.002166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.002084 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm\" (UniqueName: \"kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm\") pod \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " Oct 31 03:40:24.002399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.002240 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca\") pod \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\" (UID: \"5605ed00-fdcc-4d12-8355-1fbc8e4ba903\") " Oct 31 03:40:24.002505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.002473 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca" pod "5605ed00-fdcc-4d12-8355-1fbc8e4ba903" (UID: "5605ed00-fdcc-4d12-8355-1fbc8e4ba903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:24.002812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.002786 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:24.004398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.004369 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca" pod "5605ed00-fdcc-4d12-8355-1fbc8e4ba903" (UID: "5605ed00-fdcc-4d12-8355-1fbc8e4ba903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:24.004532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.004511 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm" pod "5605ed00-fdcc-4d12-8355-1fbc8e4ba903" (UID: "5605ed00-fdcc-4d12-8355-1fbc8e4ba903"). InnerVolumeSpecName "kube-api-access-jbvtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:24.104672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.104633 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvtm\" (UniqueName: \"kubernetes.io/projected/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-kube-api-access-jbvtm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:24.104672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.104667 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605ed00-fdcc-4d12-8355-1fbc8e4ba903-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:24.741395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.741365 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" Oct 31 03:40:24.741602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.741401 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj" event={"ID":"5605ed00-fdcc-4d12-8355-1fbc8e4ba903","Type":"ContainerDied","Data":"214427464dadc5c02f75c07568bc0ba1d0a0b033654814e9fa33f958e5e25c19"} Oct 31 03:40:24.741602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:24.741452 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="214427464dadc5c02f75c07568bc0ba1d0a0b033654814e9fa33f958e5e25c19" Oct 31 03:40:26.166036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166000 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc"] Oct 31 03:40:26.166562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166546 4488 cpu_manager.go:414] "containerMap: removing container" podUID="5605ed00-fdcc-4d12-8355-1fbc8e4ba903" containerName="revoke-keys" Oct 31 03:40:26.166609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166562 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5605ed00-fdcc-4d12-8355-1fbc8e4ba903" containerName="revoke-keys" Oct 31 03:40:26.166609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166578 4488 cpu_manager.go:414] "containerMap: removing container" podUID="bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" containerName="revoke-keys" Oct 31 03:40:26.166609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166583 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" containerName="revoke-keys" Oct 31 03:40:26.166716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166707 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="5605ed00-fdcc-4d12-8355-1fbc8e4ba903" containerName="revoke-keys" Oct 31 03:40:26.166755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.166718 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" containerName="revoke-keys" Oct 31 03:40:26.170418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.170402 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.272994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.272954 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:40:26.272994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.272972 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:40:26.272994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.272959 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:40:26.273281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.273042 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:40:26.277012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.276987 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc"] Oct 31 03:40:26.333598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.333571 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.333705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.333636 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.333705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.333654 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqknh\" (UniqueName: \"kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.377605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.377575 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9"] Oct 31 03:40:26.381676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.381654 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.434979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.434886 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435020 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435154 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435190 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnpjn\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435307 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435284 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435319 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435407 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435480 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.435541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435517 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.435642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435602 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.435778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435665 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqknh\" (UniqueName: \"kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.435778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.435739 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.436451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.436424 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.438540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.438517 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.480748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.480705 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9"] Oct 31 03:40:26.497505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.497476 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqknh\" (UniqueName: \"kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh\") pod \"maas-api-revoke-sub-04d9b98e-mgfxc\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:26.536533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536499 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536602 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536657 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536736 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536783 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536808 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpjn\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536852 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536882 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.536980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536944 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.537190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.536943 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.537256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.537215 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.537305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.537225 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.537341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.537316 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.537538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.537510 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.539077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.539055 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.539457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.539442 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.598682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.598642 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.598838 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.598751 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpjn\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.695592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.695511 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:26.788554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:26.788272 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:27.135083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.135056 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9"] Oct 31 03:40:27.135688 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:27.135646 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726d0934_3c43_46dc_8b27_be9a0dc2ac71.slice/crio-c13d75ccc27179a5c097f73d1e712b2566c43d53599cb876c9f16ba89edbd107 WatchSource:0}: Error finding container c13d75ccc27179a5c097f73d1e712b2566c43d53599cb876c9f16ba89edbd107: Status 404 returned error can't find the container with id c13d75ccc27179a5c097f73d1e712b2566c43d53599cb876c9f16ba89edbd107 Oct 31 03:40:27.138239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.138205 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:27.138320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.138273 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:27.138320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.138304 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:27.186958 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:27.186929 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fdcf82_327a_4770_82e1_2e56d89ddcc7.slice/crio-ab86fa7df3f135fd8c6027af38020f05d561f3bcac82304426c30a12fdf91a25 WatchSource:0}: Error finding container ab86fa7df3f135fd8c6027af38020f05d561f3bcac82304426c30a12fdf91a25: Status 404 returned error can't find the container with id ab86fa7df3f135fd8c6027af38020f05d561f3bcac82304426c30a12fdf91a25 Oct 31 03:40:27.187780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.187752 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc"] Oct 31 03:40:27.819185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.818108 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" event={"ID":"11fdcf82-327a-4770-82e1-2e56d89ddcc7","Type":"ContainerStarted","Data":"57150e8f76dff86e36e115b976062bef567edca763b6ec045e92cb23d16037f9"} Oct 31 03:40:27.819185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.818178 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" event={"ID":"11fdcf82-327a-4770-82e1-2e56d89ddcc7","Type":"ContainerStarted","Data":"ab86fa7df3f135fd8c6027af38020f05d561f3bcac82304426c30a12fdf91a25"} Oct 31 03:40:27.823111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.822396 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" event={"ID":"726d0934-3c43-46dc-8b27-be9a0dc2ac71","Type":"ContainerStarted","Data":"cccdfc70a2829abc1c6a7195f898a1c1e1c7921a113e984298044bc389525b88"} Oct 31 03:40:27.823111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.822502 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" event={"ID":"726d0934-3c43-46dc-8b27-be9a0dc2ac71","Type":"ContainerStarted","Data":"c13d75ccc27179a5c097f73d1e712b2566c43d53599cb876c9f16ba89edbd107"} Oct 31 03:40:27.838396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.838356 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:27.948225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:27.948159 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" podStartSLOduration=1.948135267 podStartE2EDuration="1.948135267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:27.934816232 +0000 UTC m=+1255.731534491" watchObservedRunningTime="2026-10-31 03:40:27.948135267 +0000 UTC m=+1255.744853526" Oct 31 03:40:28.055657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:28.055313 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podStartSLOduration=2.055291474 podStartE2EDuration="2.055291474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:28.054808651 +0000 UTC m=+1255.851526938" watchObservedRunningTime="2026-10-31 03:40:28.055291474 +0000 UTC m=+1255.852009735" Oct 31 03:40:28.696032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:28.696002 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:28.697529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:28.697509 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:28.697600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:28.697555 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:28.838463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:28.838425 4488 generic.go:356] "Generic (PLEG): container finished" podID="11fdcf82-327a-4770-82e1-2e56d89ddcc7" containerID="57150e8f76dff86e36e115b976062bef567edca763b6ec045e92cb23d16037f9" exitCode=0 Oct 31 03:40:28.838655 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:28.838513 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" event={"ID":"11fdcf82-327a-4770-82e1-2e56d89ddcc7","Type":"ContainerDied","Data":"57150e8f76dff86e36e115b976062bef567edca763b6ec045e92cb23d16037f9"} Oct 31 03:40:29.050730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.050637 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx"] Oct 31 03:40:29.054396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.054365 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c10b9bf-8gljx"] Oct 31 03:40:29.537345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.537310 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fd8mr-np954"] Oct 31 03:40:29.541420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.541401 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.577594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.577562 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.577780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.577756 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.577842 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.577820 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtss\" (UniqueName: \"kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.577901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.577883 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.594013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.593984 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fd8mr-np954"] Oct 31 03:40:29.594153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.594079 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 03:40:29.639244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.639215 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jjf7q\"" Oct 31 03:40:29.678942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.678910 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtss\" (UniqueName: \"kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.678942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.678947 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.679243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.679150 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.679315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.679298 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.679504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.679481 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.679724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.679702 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.696634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.696609 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:29.696998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.696652 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:29.744144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.744117 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 03:40:29.753153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.750039 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtss\" (UniqueName: \"kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.753153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.750155 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle\") pod \"periodic-gathering-fd8mr-np954\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.773135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.773058 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz"] Oct 31 03:40:29.782137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.782118 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.785538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.785506 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz"] Oct 31 03:40:29.854084 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.854061 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:40:29.885290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.885255 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.885447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.885380 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.885515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.885455 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj2m\" (UniqueName: \"kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.987299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.987264 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.987454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.987371 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.987454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.987432 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj2m\" (UniqueName: \"kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.989049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.989019 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:29.990667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:29.990643 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:30.052524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.052496 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj2m\" (UniqueName: \"kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m\") pod \"maas-api-revoke-sub-42ab031f-nlccz\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:30.081422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.081397 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:30.097892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.097863 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:30.195140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.189710 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh\" (UniqueName: \"kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh\") pod \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " Oct 31 03:40:30.195140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.189989 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca\") pod \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " Oct 31 03:40:30.195140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.190121 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca\") pod \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\" (UID: \"11fdcf82-327a-4770-82e1-2e56d89ddcc7\") " Oct 31 03:40:30.195140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.190996 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca" pod "11fdcf82-327a-4770-82e1-2e56d89ddcc7" (UID: "11fdcf82-327a-4770-82e1-2e56d89ddcc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:30.195140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.193661 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh" pod "11fdcf82-327a-4770-82e1-2e56d89ddcc7" (UID: "11fdcf82-327a-4770-82e1-2e56d89ddcc7"). InnerVolumeSpecName "kube-api-access-kqknh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:30.195634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.195607 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca" pod "11fdcf82-327a-4770-82e1-2e56d89ddcc7" (UID: "11fdcf82-327a-4770-82e1-2e56d89ddcc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:30.291664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.291629 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:30.291664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.291661 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fdcf82-327a-4770-82e1-2e56d89ddcc7-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:30.291664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.291672 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqknh\" (UniqueName: \"kubernetes.io/projected/11fdcf82-327a-4770-82e1-2e56d89ddcc7-kube-api-access-kqknh\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:30.334507 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:30.334470 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66682f45_69b8_4d48_8c8f_6769133ee137.slice/crio-62b9b721dc1b2d513ce6da371e3b066ad2492252c9a90e00f587379f62a7585d WatchSource:0}: Error finding container 62b9b721dc1b2d513ce6da371e3b066ad2492252c9a90e00f587379f62a7585d: Status 404 returned error can't find the container with id 62b9b721dc1b2d513ce6da371e3b066ad2492252c9a90e00f587379f62a7585d Oct 31 03:40:30.339377 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.339356 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e550453-c9ea-4bb7-af38-a41f24fa6d0f" path="/var/lib/kubelet/pods/8e550453-c9ea-4bb7-af38-a41f24fa6d0f/volumes" Oct 31 03:40:30.339887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.339871 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fd8mr-np954"] Oct 31 03:40:30.501918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.501888 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz"] Oct 31 03:40:30.502485 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:30.502456 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bbd886_2b3c_4a4c_8592_b6b8665a2e48.slice/crio-57808da3e87cd58e56da9aed28fb1963780cb45f76a80481030900f34fdd00c7 WatchSource:0}: Error finding container 57808da3e87cd58e56da9aed28fb1963780cb45f76a80481030900f34fdd00c7: Status 404 returned error can't find the container with id 57808da3e87cd58e56da9aed28fb1963780cb45f76a80481030900f34fdd00c7 Oct 31 03:40:30.696309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.696224 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:30.696309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.696285 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:30.853570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.853532 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" event={"ID":"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48","Type":"ContainerStarted","Data":"57808da3e87cd58e56da9aed28fb1963780cb45f76a80481030900f34fdd00c7"} Oct 31 03:40:30.855173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.855145 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fd8mr-np954" event={"ID":"66682f45-69b8-4d48-8c8f-6769133ee137","Type":"ContainerStarted","Data":"d790209e63e67374862b124807d37c03e2594b40582283006761e56a0aeaca4f"} Oct 31 03:40:30.855327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.855191 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fd8mr-np954" event={"ID":"66682f45-69b8-4d48-8c8f-6769133ee137","Type":"ContainerStarted","Data":"62b9b721dc1b2d513ce6da371e3b066ad2492252c9a90e00f587379f62a7585d"} Oct 31 03:40:30.856640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.856622 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" event={"ID":"11fdcf82-327a-4770-82e1-2e56d89ddcc7","Type":"ContainerDied","Data":"ab86fa7df3f135fd8c6027af38020f05d561f3bcac82304426c30a12fdf91a25"} Oct 31 03:40:30.856734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.856639 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc" Oct 31 03:40:30.856734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.856658 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab86fa7df3f135fd8c6027af38020f05d561f3bcac82304426c30a12fdf91a25" Oct 31 03:40:30.985454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:30.985336 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fd8mr-np954" podStartSLOduration=1.985317807 podStartE2EDuration="1.985317807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:30.984572047 +0000 UTC m=+1258.781290311" watchObservedRunningTime="2026-10-31 03:40:30.985317807 +0000 UTC m=+1258.782036066" Oct 31 03:40:31.054355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.054326 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n"] Oct 31 03:40:31.057977 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.057952 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9"] Oct 31 03:40:31.061065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.061043 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef41777b-8c4k9"] Oct 31 03:40:31.064157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.064136 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a977a059-zzd6n"] Oct 31 03:40:31.700120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.699391 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:31.700120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.699445 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:31.943115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.939943 4488 generic.go:356] "Generic (PLEG): container finished" podID="a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" containerID="f5e6507135bd87f55ab362620c82238f118ff6a61eb5ebd0a7530abb95c6e5d4" exitCode=0 Oct 31 03:40:31.943115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:31.941407 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" event={"ID":"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48","Type":"ContainerDied","Data":"f5e6507135bd87f55ab362620c82238f118ff6a61eb5ebd0a7530abb95c6e5d4"} Oct 31 03:40:32.493379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:32.466331 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782730d8-42b1-41db-9fd1-4086d6a95026" path="/var/lib/kubelet/pods/782730d8-42b1-41db-9fd1-4086d6a95026/volumes" Oct 31 03:40:32.493379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:32.467426 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96441321-406e-45cd-8238-d690de5e21fb" path="/var/lib/kubelet/pods/96441321-406e-45cd-8238-d690de5e21fb/volumes" Oct 31 03:40:32.726115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:32.680788 4488 scope.go:122] "RemoveContainer" containerID="fd4479d12adb47dbe034b2737d2cd6f3d4b3fb2fc9cb9a6a9a4078a40b1a2b3e" Oct 31 03:40:32.726115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:32.703737 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:32.726115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:32.703819 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:32.871997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:32.871887 4488 scope.go:122] "RemoveContainer" containerID="700475cab5a7fe19017743c98966e29208e329a33e12041fe80e130672bcd4bc" Oct 31 03:40:33.163379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.163259 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:33.186595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.186569 4488 scope.go:122] "RemoveContainer" containerID="1e271b1d0304af046fa3067bf03596b00227ce65f907732e54362466a3bb3abc" Oct 31 03:40:33.519296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.518488 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.620835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.620779 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.622601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.622568 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.659022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.658198 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.684322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.684203 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.694255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.694209 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.695521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.695475 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.696865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.696826 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:33.697102 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.697059 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:33.735314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.735249 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.740578 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.740523 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.750708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.750618 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.751573 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.751539 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.753689 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.753650 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.842815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.841379 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:33.935184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:33.933268 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:40:34.186025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.185916 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.188326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.188290 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.558254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.524706 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.558254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.525048 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:40:34.650276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.638016 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.650276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.642928 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.650276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.646588 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:40:34.650276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.646746 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:40:34.677566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.663224 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.677566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.664543 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:40:34.708046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.691773 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:40:34.708046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.692246 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:40:34.714555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.714509 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.732529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.720182 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.732529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.721413 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:40:34.732529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.721481 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:40:34.750122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.748236 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.750122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.748566 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.750122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.748813 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:40:34.750122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.748871 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:40:34.802104 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.800973 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.803111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.802846 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:40:34.831119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.803415 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:40:34.831119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.805784 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:40:34.851836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.846676 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.851836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.848653 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:40:34.851836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.849661 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:34.851836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.849707 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.851836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.850616 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.892117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.866507 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:40:34.892117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.867293 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:34.892117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.867337 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:34.892117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.871122 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Oct 31 03:40:34.956005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.937374 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:34.959910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:34.959857 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:40:35.156363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.156326 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:35.247261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.247208 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.249390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.249359 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:40:35.439180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.430622 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m\" (UniqueName: \"kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m\") pod \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " Oct 31 03:40:35.466746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.466711 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca\") pod \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " Oct 31 03:40:35.502119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.472184 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca\") pod \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\" (UID: \"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48\") " Oct 31 03:40:35.557133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.466445 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:35.557133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.511278 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.557133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.511531 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca" pod "a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" (UID: "a0bbd886-2b3c-4a4c-8592-b6b8665a2e48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:35.584118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.558670 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.903117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.610437 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:35.903882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.655518 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.904448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.655573 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:35.948442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.666938 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.015650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.719217 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:36.015888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.015858 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:36.016026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.729407 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.016241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.760190 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.016348 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.760232 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:40:36.016447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.810265 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:40:36.016542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.810689 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.041231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.848390 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.041495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.864422 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.041609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.874190 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.041710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.947211 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.041818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.948300 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" event={"ID":"a0bbd886-2b3c-4a4c-8592-b6b8665a2e48","Type":"ContainerDied","Data":"57808da3e87cd58e56da9aed28fb1963780cb45f76a80481030900f34fdd00c7"} Oct 31 03:40:36.041971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.041951 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57808da3e87cd58e56da9aed28fb1963780cb45f76a80481030900f34fdd00c7" Oct 31 03:40:36.042074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:35.719258 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:40:36.099119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.066994 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz" Oct 31 03:40:36.099119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.078932 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca" pod "a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" (UID: "a0bbd886-2b3c-4a4c-8592-b6b8665a2e48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:36.159401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.159315 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:36.273119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.272378 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:36.319118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.297240 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.319118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.297731 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.997564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.971760 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:36.997564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:36.972745 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:37.028129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.014659 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:37.077119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.017616 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:40:37.503117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.457969 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 03:40:37.765547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.765275 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 31 03:40:37.768137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.054673 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:40:37.769081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.054796 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:40:37.770395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.054906 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:40:37.771804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.055037 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:40:37.772550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.055207 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:40:37.796214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.177998 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:37.815572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.804416 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:40:37.815572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.805134 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:40:37.818395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.272409 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:40:37.819405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.456973 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:40:37.819825 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:40:37.457779 4488 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.119s" Oct 31 03:40:37.820209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.820189 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7"] Oct 31 03:40:37.820389 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.820376 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffab464f-ppjp7"] Oct 31 03:40:37.820499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.017997 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m" pod "a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" (UID: "a0bbd886-2b3c-4a4c-8592-b6b8665a2e48"). InnerVolumeSpecName "kube-api-access-dkj2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:37.820656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.690522 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:37.820874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.820817 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:37.821197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.335264 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 03:40:37.821345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.051998 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:37.838774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.272182 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:40:37.844236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.844192 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:37.861657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:37.054499 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:38.110239 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:40:38.090759 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:40:38.127284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.125430 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:38.127284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.125819 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:40:38.191480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.191442 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkj2m\" (UniqueName: \"kubernetes.io/projected/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48-kube-api-access-dkj2m\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:38.212112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.208725 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:40:38.217114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.216776 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:40:38.217114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.216845 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:40:38.221852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.221824 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:40:38.223479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.223454 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:40:38.229125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.225391 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:40:38.237108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.233673 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:40:38.237108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.234747 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:40:38.237108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.235695 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:40:38.237108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.236667 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:40:38.253128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.251486 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:40:38.277134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.265589 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:40:38.294895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.294867 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:40:38.308870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.308844 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:40:38.340470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.340434 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312" path="/var/lib/kubelet/pods/374c30dc-c7ca-4ec7-b2c6-7fbb3dd94312/volumes" Oct 31 03:40:38.342028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.342008 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:40:38.664254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.664218 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:38.752471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.752431 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:38.753711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.752496 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:38.799398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.799360 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz"] Oct 31 03:40:38.800129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.800069 4488 cpu_manager.go:414] "containerMap: removing container" podUID="11fdcf82-327a-4770-82e1-2e56d89ddcc7" containerName="revoke-keys" Oct 31 03:40:38.800129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.800107 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fdcf82-327a-4770-82e1-2e56d89ddcc7" containerName="revoke-keys" Oct 31 03:40:38.800292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.800134 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" containerName="revoke-keys" Oct 31 03:40:38.800292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.800142 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" containerName="revoke-keys" Oct 31 03:40:38.800292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.800273 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fdcf82-327a-4770-82e1-2e56d89ddcc7" containerName="revoke-keys" Oct 31 03:40:38.800292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.800284 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" containerName="revoke-keys" Oct 31 03:40:38.805063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.805035 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:38.859034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.859002 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:40:38.859546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.859024 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:40:38.859627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.859209 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:40:38.859627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.859274 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:40:38.862653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.859921 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d"] Oct 31 03:40:38.864958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.864932 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:38.865324 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.865301 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz"] Oct 31 03:40:38.911913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.911863 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:38.912130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.912047 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:38.912130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.912074 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:38.912274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.912214 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6sjf\" (UniqueName: \"kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:38.912356 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.912330 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hzd\" (UniqueName: \"kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:38.912421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.912402 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:38.932781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:38.932590 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d"] Oct 31 03:40:39.013828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.013776 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6sjf\" (UniqueName: \"kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.014042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.013861 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hzd\" (UniqueName: \"kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.014042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.013916 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.014219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.014197 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.014272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.014259 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.014331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.014284 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.014947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.014916 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.015138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.015042 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.032119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.017272 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.032119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.025830 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.043145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.038783 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm"] Oct 31 03:40:39.051818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.047546 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.092930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.085889 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hzd\" (UniqueName: \"kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd\") pod \"maas-api-revoke-sub-d325dc96-72q5d\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.092930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.089584 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4"] Oct 31 03:40:39.096393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.096223 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6sjf\" (UniqueName: \"kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf\") pod \"maas-api-revoke-sub-10c18149-6smrz\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.097716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.097687 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ef050d-cbnz4"] Oct 31 03:40:39.104422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.103877 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm"] Oct 31 03:40:39.117018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.116924 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.117618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.117591 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zc6\" (UniqueName: \"kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.117883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.117863 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.132330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.132271 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:39.189621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.189511 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:39.229194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.229149 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.229982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.229960 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zc6\" (UniqueName: \"kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.230555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.230533 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.231475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.231372 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.245559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.236568 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.292795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.292720 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zc6\" (UniqueName: \"kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6\") pod \"maas-api-revoke-sub-dcb70e33-tzplm\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.473746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.473664 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:39.606266 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:39.606237 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c3f234_ecdc_4739_a6e7_de4ee34c65fd.slice/crio-ccf6d890c06fae12ca3e9210b8be70c5b2b43ea46bc3e41caf2d1726c516bd49 WatchSource:0}: Error finding container ccf6d890c06fae12ca3e9210b8be70c5b2b43ea46bc3e41caf2d1726c516bd49: Status 404 returned error can't find the container with id ccf6d890c06fae12ca3e9210b8be70c5b2b43ea46bc3e41caf2d1726c516bd49 Oct 31 03:40:39.609366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.609339 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz"] Oct 31 03:40:39.661630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.661607 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d"] Oct 31 03:40:39.666130 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:39.666073 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133c1810_55c4_484b_9d1d_da01cc0084bb.slice/crio-807b3d759a036d9daff325c0abff9ef685ba810be8649b51e46602d8024802e1 WatchSource:0}: Error finding container 807b3d759a036d9daff325c0abff9ef685ba810be8649b51e46602d8024802e1: Status 404 returned error can't find the container with id 807b3d759a036d9daff325c0abff9ef685ba810be8649b51e46602d8024802e1 Oct 31 03:40:39.753122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.752993 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:39.753285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.753070 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:39.818048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.818009 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:39.818460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.818435 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:39.819502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.819480 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:40:39.905287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:39.905260 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm"] Oct 31 03:40:39.907829 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:39.907798 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f49d4c_3032_4885_b74d_d6326c92fae3.slice/crio-39b23285ca0eab1505e01f36844d444e39c240850dbca5d7b1fd2b349d497307 WatchSource:0}: Error finding container 39b23285ca0eab1505e01f36844d444e39c240850dbca5d7b1fd2b349d497307: Status 404 returned error can't find the container with id 39b23285ca0eab1505e01f36844d444e39c240850dbca5d7b1fd2b349d497307 Oct 31 03:40:40.161537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.161497 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" event={"ID":"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd","Type":"ContainerStarted","Data":"9dd2ccc44aa4c52e937ebb8daef96e24a2f61c9510a2e130ded6dbc369cee65d"} Oct 31 03:40:40.161711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.161552 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" event={"ID":"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd","Type":"ContainerStarted","Data":"ccf6d890c06fae12ca3e9210b8be70c5b2b43ea46bc3e41caf2d1726c516bd49"} Oct 31 03:40:40.162965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.162931 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" event={"ID":"11f49d4c-3032-4885-b74d-d6326c92fae3","Type":"ContainerStarted","Data":"39b23285ca0eab1505e01f36844d444e39c240850dbca5d7b1fd2b349d497307"} Oct 31 03:40:40.164344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.164323 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" event={"ID":"133c1810-55c4-484b-9d1d-da01cc0084bb","Type":"ContainerStarted","Data":"807b3d759a036d9daff325c0abff9ef685ba810be8649b51e46602d8024802e1"} Oct 31 03:40:40.323178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.323073 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4533698d-250f-4cb7-9b7c-711174882642" path="/var/lib/kubelet/pods/4533698d-250f-4cb7-9b7c-711174882642/volumes" Oct 31 03:40:40.660841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.660781 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" podStartSLOduration=2.660767353 podStartE2EDuration="2.660767353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:40.657270337 +0000 UTC m=+1268.453988610" watchObservedRunningTime="2026-10-31 03:40:40.660767353 +0000 UTC m=+1268.457485647" Oct 31 03:40:40.824784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.824746 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c"] Oct 31 03:40:40.828968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.828949 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.878923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.878885 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.879122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.878943 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vh22\" (UniqueName: \"kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.879173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.879125 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.887961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.887912 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c"] Oct 31 03:40:40.980805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.980770 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.981002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.980838 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vh22\" (UniqueName: \"kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.981002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.980968 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.981778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.981754 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:40.983502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:40.983481 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:41.043769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.043737 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vh22\" (UniqueName: \"kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22\") pod \"maas-api-revoke-sub-629191e4-nxl6c\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:41.167771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.167738 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:41.173213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.173182 4488 generic.go:356] "Generic (PLEG): container finished" podID="d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" containerID="9dd2ccc44aa4c52e937ebb8daef96e24a2f61c9510a2e130ded6dbc369cee65d" exitCode=0 Oct 31 03:40:41.173336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.173256 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" event={"ID":"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd","Type":"ContainerDied","Data":"9dd2ccc44aa4c52e937ebb8daef96e24a2f61c9510a2e130ded6dbc369cee65d"} Oct 31 03:40:41.175036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.175012 4488 generic.go:356] "Generic (PLEG): container finished" podID="11f49d4c-3032-4885-b74d-d6326c92fae3" containerID="1ce047eb6c0e5322c5ef2ceddeced77d59d78ba8d993e94164faa82920a18c77" exitCode=0 Oct 31 03:40:41.175160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.175073 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" event={"ID":"11f49d4c-3032-4885-b74d-d6326c92fae3","Type":"ContainerDied","Data":"1ce047eb6c0e5322c5ef2ceddeced77d59d78ba8d993e94164faa82920a18c77"} Oct 31 03:40:41.176642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.176620 4488 generic.go:356] "Generic (PLEG): container finished" podID="133c1810-55c4-484b-9d1d-da01cc0084bb" containerID="16cb9ab86b439d087ee1274ff68acf42f5914ffc36e85ee158f27ebb96c6ee7d" exitCode=0 Oct 31 03:40:41.176746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.176691 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" event={"ID":"133c1810-55c4-484b-9d1d-da01cc0084bb","Type":"ContainerDied","Data":"16cb9ab86b439d087ee1274ff68acf42f5914ffc36e85ee158f27ebb96c6ee7d"} Oct 31 03:40:41.554453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:41.554424 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c"] Oct 31 03:40:41.555714 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:41.555676 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf86bcd_a193_4af1_b42d_8efa7c23789d.slice/crio-da40fe2ad2d33eb68577806e8be006b9cd514140b1f57872c5d506e4311a3275 WatchSource:0}: Error finding container da40fe2ad2d33eb68577806e8be006b9cd514140b1f57872c5d506e4311a3275: Status 404 returned error can't find the container with id da40fe2ad2d33eb68577806e8be006b9cd514140b1f57872c5d506e4311a3275 Oct 31 03:40:42.185608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.185566 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" event={"ID":"4bf86bcd-a193-4af1-b42d-8efa7c23789d","Type":"ContainerStarted","Data":"b936d09c0d0a58c5a58150f2bf09c17fbf9ce2d801178488ba980f5ec4b0903b"} Oct 31 03:40:42.186060 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.185633 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" event={"ID":"4bf86bcd-a193-4af1-b42d-8efa7c23789d","Type":"ContainerStarted","Data":"da40fe2ad2d33eb68577806e8be006b9cd514140b1f57872c5d506e4311a3275"} Oct 31 03:40:42.302057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.301999 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" podStartSLOduration=2.301980909 podStartE2EDuration="2.301980909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:42.301752778 +0000 UTC m=+1270.098471038" watchObservedRunningTime="2026-10-31 03:40:42.301980909 +0000 UTC m=+1270.098699167" Oct 31 03:40:42.381344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.381313 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:42.459720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.459695 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:42.463182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.463160 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:42.509834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.509803 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd\" (UniqueName: \"kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd\") pod \"133c1810-55c4-484b-9d1d-da01cc0084bb\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " Oct 31 03:40:42.509995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.509976 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6\" (UniqueName: \"kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6\") pod \"11f49d4c-3032-4885-b74d-d6326c92fae3\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " Oct 31 03:40:42.510063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510015 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca\") pod \"11f49d4c-3032-4885-b74d-d6326c92fae3\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " Oct 31 03:40:42.510164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510079 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca\") pod \"133c1810-55c4-484b-9d1d-da01cc0084bb\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " Oct 31 03:40:42.510164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510132 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca\") pod \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " Oct 31 03:40:42.510272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510175 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca\") pod \"11f49d4c-3032-4885-b74d-d6326c92fae3\" (UID: \"11f49d4c-3032-4885-b74d-d6326c92fae3\") " Oct 31 03:40:42.510322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510274 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca\") pod \"133c1810-55c4-484b-9d1d-da01cc0084bb\" (UID: \"133c1810-55c4-484b-9d1d-da01cc0084bb\") " Oct 31 03:40:42.510375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510329 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca\") pod \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " Oct 31 03:40:42.510433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510413 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf\" (UniqueName: \"kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf\") pod \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\" (UID: \"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd\") " Oct 31 03:40:42.510886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510502 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca" pod "11f49d4c-3032-4885-b74d-d6326c92fae3" (UID: "11f49d4c-3032-4885-b74d-d6326c92fae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:42.510886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510523 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca" pod "133c1810-55c4-484b-9d1d-da01cc0084bb" (UID: "133c1810-55c4-484b-9d1d-da01cc0084bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:42.510886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.510841 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca" pod "d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" (UID: "d5c3f234-ecdc-4739-a6e7-de4ee34c65fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:42.511159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.511038 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.511159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.511061 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.511159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.511075 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.513832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.513802 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca" pod "11f49d4c-3032-4885-b74d-d6326c92fae3" (UID: "11f49d4c-3032-4885-b74d-d6326c92fae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:42.513955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.513865 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf" pod "d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" (UID: "d5c3f234-ecdc-4739-a6e7-de4ee34c65fd"). InnerVolumeSpecName "kube-api-access-f6sjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:42.514447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.514394 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca" pod "133c1810-55c4-484b-9d1d-da01cc0084bb" (UID: "133c1810-55c4-484b-9d1d-da01cc0084bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:42.514575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.514552 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd" pod "133c1810-55c4-484b-9d1d-da01cc0084bb" (UID: "133c1810-55c4-484b-9d1d-da01cc0084bb"). InnerVolumeSpecName "kube-api-access-v8hzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:42.514722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.514700 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca" pod "d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" (UID: "d5c3f234-ecdc-4739-a6e7-de4ee34c65fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:42.515790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.515770 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6" pod "11f49d4c-3032-4885-b74d-d6326c92fae3" (UID: "11f49d4c-3032-4885-b74d-d6326c92fae3"). InnerVolumeSpecName "kube-api-access-t4zc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:42.613300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.613263 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6sjf\" (UniqueName: \"kubernetes.io/projected/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-kube-api-access-f6sjf\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.613300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.613302 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hzd\" (UniqueName: \"kubernetes.io/projected/133c1810-55c4-484b-9d1d-da01cc0084bb-kube-api-access-v8hzd\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.613503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.613316 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4zc6\" (UniqueName: \"kubernetes.io/projected/11f49d4c-3032-4885-b74d-d6326c92fae3-kube-api-access-t4zc6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.613503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.613330 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f49d4c-3032-4885-b74d-d6326c92fae3-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.613503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.613346 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133c1810-55c4-484b-9d1d-da01cc0084bb-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:42.613503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:42.613359 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:43.195136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.195074 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" Oct 31 03:40:43.195540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.195139 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz" event={"ID":"d5c3f234-ecdc-4739-a6e7-de4ee34c65fd","Type":"ContainerDied","Data":"ccf6d890c06fae12ca3e9210b8be70c5b2b43ea46bc3e41caf2d1726c516bd49"} Oct 31 03:40:43.195540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.195212 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf6d890c06fae12ca3e9210b8be70c5b2b43ea46bc3e41caf2d1726c516bd49" Oct 31 03:40:43.196907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.196879 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" Oct 31 03:40:43.197044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.196874 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm" event={"ID":"11f49d4c-3032-4885-b74d-d6326c92fae3","Type":"ContainerDied","Data":"39b23285ca0eab1505e01f36844d444e39c240850dbca5d7b1fd2b349d497307"} Oct 31 03:40:43.197044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.197012 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39b23285ca0eab1505e01f36844d444e39c240850dbca5d7b1fd2b349d497307" Oct 31 03:40:43.198446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.198424 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" event={"ID":"133c1810-55c4-484b-9d1d-da01cc0084bb","Type":"ContainerDied","Data":"807b3d759a036d9daff325c0abff9ef685ba810be8649b51e46602d8024802e1"} Oct 31 03:40:43.198567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.198465 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807b3d759a036d9daff325c0abff9ef685ba810be8649b51e46602d8024802e1" Oct 31 03:40:43.198567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.198434 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d" Oct 31 03:40:43.200138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.200080 4488 generic.go:356] "Generic (PLEG): container finished" podID="4bf86bcd-a193-4af1-b42d-8efa7c23789d" containerID="b936d09c0d0a58c5a58150f2bf09c17fbf9ce2d801178488ba980f5ec4b0903b" exitCode=0 Oct 31 03:40:43.200234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.200178 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" event={"ID":"4bf86bcd-a193-4af1-b42d-8efa7c23789d","Type":"ContainerDied","Data":"b936d09c0d0a58c5a58150f2bf09c17fbf9ce2d801178488ba980f5ec4b0903b"} Oct 31 03:40:43.440791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:43.440756 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:44.375395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.375368 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:44.437604 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.437569 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca\") pod \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " Oct 31 03:40:44.437784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.437650 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22\" (UniqueName: \"kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22\") pod \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " Oct 31 03:40:44.437784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.437717 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca\") pod \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\" (UID: \"4bf86bcd-a193-4af1-b42d-8efa7c23789d\") " Oct 31 03:40:44.437994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.437970 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca" pod "4bf86bcd-a193-4af1-b42d-8efa7c23789d" (UID: "4bf86bcd-a193-4af1-b42d-8efa7c23789d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:44.438309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.438287 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:44.439998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.439976 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22" pod "4bf86bcd-a193-4af1-b42d-8efa7c23789d" (UID: "4bf86bcd-a193-4af1-b42d-8efa7c23789d"). InnerVolumeSpecName "kube-api-access-2vh22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:44.440066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.440000 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca" pod "4bf86bcd-a193-4af1-b42d-8efa7c23789d" (UID: "4bf86bcd-a193-4af1-b42d-8efa7c23789d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:44.539565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.539484 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vh22\" (UniqueName: \"kubernetes.io/projected/4bf86bcd-a193-4af1-b42d-8efa7c23789d-kube-api-access-2vh22\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:44.539565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.539514 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf86bcd-a193-4af1-b42d-8efa7c23789d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:44.630834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.630796 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq"] Oct 31 03:40:44.631432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631413 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631433 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631466 4488 cpu_manager.go:414] "containerMap: removing container" podUID="11f49d4c-3032-4885-b74d-d6326c92fae3" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631473 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f49d4c-3032-4885-b74d-d6326c92fae3" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631482 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4bf86bcd-a193-4af1-b42d-8efa7c23789d" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631488 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf86bcd-a193-4af1-b42d-8efa7c23789d" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631519 4488 cpu_manager.go:414] "containerMap: removing container" podUID="133c1810-55c4-484b-9d1d-da01cc0084bb" containerName="revoke-keys" Oct 31 03:40:44.631555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631526 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133c1810-55c4-484b-9d1d-da01cc0084bb" containerName="revoke-keys" Oct 31 03:40:44.631871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631666 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" containerName="revoke-keys" Oct 31 03:40:44.631871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631682 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="133c1810-55c4-484b-9d1d-da01cc0084bb" containerName="revoke-keys" Oct 31 03:40:44.631871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631697 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf86bcd-a193-4af1-b42d-8efa7c23789d" containerName="revoke-keys" Oct 31 03:40:44.631871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.631706 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f49d4c-3032-4885-b74d-d6326c92fae3" containerName="revoke-keys" Oct 31 03:40:44.635932 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.635909 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.693258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.693223 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq"] Oct 31 03:40:44.743019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.742973 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.743213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.743034 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.743213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.743134 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6kc\" (UniqueName: \"kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.844563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.844525 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.844563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.844569 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6kc\" (UniqueName: \"kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.844883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.844866 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.845273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.845248 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.847373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.847346 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.906310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.906279 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6kc\" (UniqueName: \"kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc\") pod \"maas-api-revoke-sub-e0b50722-4rzbq\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:44.950489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:44.950460 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:45.219015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.218940 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" Oct 31 03:40:45.219015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.218966 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c" event={"ID":"4bf86bcd-a193-4af1-b42d-8efa7c23789d","Type":"ContainerDied","Data":"da40fe2ad2d33eb68577806e8be006b9cd514140b1f57872c5d506e4311a3275"} Oct 31 03:40:45.219220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.219020 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da40fe2ad2d33eb68577806e8be006b9cd514140b1f57872c5d506e4311a3275" Oct 31 03:40:45.341229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.341202 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq"] Oct 31 03:40:45.341460 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:45.341430 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe68c202_5b09_4072_b6ad_8437b992cb08.slice/crio-d1c9840d3106cf90f8128d09b517a465d3f50cba04829848646f63d5155d2ab9 WatchSource:0}: Error finding container d1c9840d3106cf90f8128d09b517a465d3f50cba04829848646f63d5155d2ab9: Status 404 returned error can't find the container with id d1c9840d3106cf90f8128d09b517a465d3f50cba04829848646f63d5155d2ab9 Oct 31 03:40:45.354975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.354951 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l"] Oct 31 03:40:45.360268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.360246 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.453989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.453944 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjm5\" (UniqueName: \"kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.453989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.453974 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l"] Oct 31 03:40:45.454526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.454282 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.454526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.454323 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.555824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.555789 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjm5\" (UniqueName: \"kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.555989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.555932 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.555989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.555975 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.556734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.556716 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.558499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.558479 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.617010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.616853 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjm5\" (UniqueName: \"kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5\") pod \"maas-api-revoke-sub-65d50d52-8gg4l\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:45.703058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:45.703020 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:46.051407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.051371 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7"] Oct 31 03:40:46.056553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.056528 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55548e9-8p4d7"] Oct 31 03:40:46.098789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.098758 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l"] Oct 31 03:40:46.122953 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:46.122922 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652a02e6_d553_4394_9e13_46a916638f52.slice/crio-e5ed0801a506a0450ae1dfd79efd3da34f926275203f45c6b76d9baf31db1319 WatchSource:0}: Error finding container e5ed0801a506a0450ae1dfd79efd3da34f926275203f45c6b76d9baf31db1319: Status 404 returned error can't find the container with id e5ed0801a506a0450ae1dfd79efd3da34f926275203f45c6b76d9baf31db1319 Oct 31 03:40:46.229879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.229844 4488 generic.go:356] "Generic (PLEG): container finished" podID="fe68c202-5b09-4072-b6ad-8437b992cb08" containerID="6a21c37e4eb7a9142350491b2e440456072eba2552bc6610294b9662d9af22f2" exitCode=0 Oct 31 03:40:46.230078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.229933 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" event={"ID":"fe68c202-5b09-4072-b6ad-8437b992cb08","Type":"ContainerDied","Data":"6a21c37e4eb7a9142350491b2e440456072eba2552bc6610294b9662d9af22f2"} Oct 31 03:40:46.230078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.229993 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" event={"ID":"fe68c202-5b09-4072-b6ad-8437b992cb08","Type":"ContainerStarted","Data":"d1c9840d3106cf90f8128d09b517a465d3f50cba04829848646f63d5155d2ab9"} Oct 31 03:40:46.231421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.231397 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" event={"ID":"652a02e6-d553-4394-9e13-46a916638f52","Type":"ContainerStarted","Data":"e5ed0801a506a0450ae1dfd79efd3da34f926275203f45c6b76d9baf31db1319"} Oct 31 03:40:46.317363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:46.317283 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73b3f32-3340-4e62-9bac-20284e921d61" path="/var/lib/kubelet/pods/b73b3f32-3340-4e62-9bac-20284e921d61/volumes" Oct 31 03:40:47.055883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.055850 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw"] Oct 31 03:40:47.059048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.059022 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7a03d-qjxhw"] Oct 31 03:40:47.240808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.240772 4488 generic.go:356] "Generic (PLEG): container finished" podID="652a02e6-d553-4394-9e13-46a916638f52" containerID="fc5cc877450bcb0f4ffc57d85f0f355303e092fd37f1ceea419ea48be2829692" exitCode=0 Oct 31 03:40:47.241021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.240857 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" event={"ID":"652a02e6-d553-4394-9e13-46a916638f52","Type":"ContainerDied","Data":"fc5cc877450bcb0f4ffc57d85f0f355303e092fd37f1ceea419ea48be2829692"} Oct 31 03:40:47.412998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.412971 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:47.479678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.479644 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc\" (UniqueName: \"kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc\") pod \"fe68c202-5b09-4072-b6ad-8437b992cb08\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " Oct 31 03:40:47.479914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.479872 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca\") pod \"fe68c202-5b09-4072-b6ad-8437b992cb08\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " Oct 31 03:40:47.480013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.479997 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca\") pod \"fe68c202-5b09-4072-b6ad-8437b992cb08\" (UID: \"fe68c202-5b09-4072-b6ad-8437b992cb08\") " Oct 31 03:40:47.480413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.480389 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca" pod "fe68c202-5b09-4072-b6ad-8437b992cb08" (UID: "fe68c202-5b09-4072-b6ad-8437b992cb08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:47.480652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.480632 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:47.482680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.482655 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca" pod "fe68c202-5b09-4072-b6ad-8437b992cb08" (UID: "fe68c202-5b09-4072-b6ad-8437b992cb08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:47.482802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.482703 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc" pod "fe68c202-5b09-4072-b6ad-8437b992cb08" (UID: "fe68c202-5b09-4072-b6ad-8437b992cb08"). InnerVolumeSpecName "kube-api-access-dq6kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:47.581665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.581625 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe68c202-5b09-4072-b6ad-8437b992cb08-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:47.581665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:47.581660 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6kc\" (UniqueName: \"kubernetes.io/projected/fe68c202-5b09-4072-b6ad-8437b992cb08-kube-api-access-dq6kc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.215105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.215037 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:48.265907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.265869 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" Oct 31 03:40:48.266110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.265932 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq" event={"ID":"fe68c202-5b09-4072-b6ad-8437b992cb08","Type":"ContainerDied","Data":"d1c9840d3106cf90f8128d09b517a465d3f50cba04829848646f63d5155d2ab9"} Oct 31 03:40:48.266110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.265987 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c9840d3106cf90f8128d09b517a465d3f50cba04829848646f63d5155d2ab9" Oct 31 03:40:48.322809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.322774 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00f1c3e-401f-461c-811f-71a6cf5ba1eb" path="/var/lib/kubelet/pods/c00f1c3e-401f-461c-811f-71a6cf5ba1eb/volumes" Oct 31 03:40:48.452481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.452454 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:48.596142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.596105 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca\") pod \"652a02e6-d553-4394-9e13-46a916638f52\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " Oct 31 03:40:48.596338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.596168 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5\" (UniqueName: \"kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5\") pod \"652a02e6-d553-4394-9e13-46a916638f52\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " Oct 31 03:40:48.596338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.596202 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca\") pod \"652a02e6-d553-4394-9e13-46a916638f52\" (UID: \"652a02e6-d553-4394-9e13-46a916638f52\") " Oct 31 03:40:48.596900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.596867 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca" pod "652a02e6-d553-4394-9e13-46a916638f52" (UID: "652a02e6-d553-4394-9e13-46a916638f52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:48.598426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.598401 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5" pod "652a02e6-d553-4394-9e13-46a916638f52" (UID: "652a02e6-d553-4394-9e13-46a916638f52"). InnerVolumeSpecName "kube-api-access-jtjm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:48.598515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.598421 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca" pod "652a02e6-d553-4394-9e13-46a916638f52" (UID: "652a02e6-d553-4394-9e13-46a916638f52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:48.697672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.697631 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652a02e6-d553-4394-9e13-46a916638f52-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.697672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.697661 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtjm5\" (UniqueName: \"kubernetes.io/projected/652a02e6-d553-4394-9e13-46a916638f52-kube-api-access-jtjm5\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.697672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:48.697678 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652a02e6-d553-4394-9e13-46a916638f52-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:49.089512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.089476 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m"] Oct 31 03:40:49.090012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.089999 4488 cpu_manager.go:414] "containerMap: removing container" podUID="fe68c202-5b09-4072-b6ad-8437b992cb08" containerName="revoke-keys" Oct 31 03:40:49.090069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.090013 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe68c202-5b09-4072-b6ad-8437b992cb08" containerName="revoke-keys" Oct 31 03:40:49.090069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.090025 4488 cpu_manager.go:414] "containerMap: removing container" podUID="652a02e6-d553-4394-9e13-46a916638f52" containerName="revoke-keys" Oct 31 03:40:49.090069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.090029 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652a02e6-d553-4394-9e13-46a916638f52" containerName="revoke-keys" Oct 31 03:40:49.090190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.090162 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="652a02e6-d553-4394-9e13-46a916638f52" containerName="revoke-keys" Oct 31 03:40:49.090190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.090176 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe68c202-5b09-4072-b6ad-8437b992cb08" containerName="revoke-keys" Oct 31 03:40:49.093623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.093596 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.151408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.151369 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m"] Oct 31 03:40:49.205270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.205232 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.205418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.205401 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9dk\" (UniqueName: \"kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.205470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.205459 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.275444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.275417 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" Oct 31 03:40:49.275801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.275409 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l" event={"ID":"652a02e6-d553-4394-9e13-46a916638f52","Type":"ContainerDied","Data":"e5ed0801a506a0450ae1dfd79efd3da34f926275203f45c6b76d9baf31db1319"} Oct 31 03:40:49.275801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.275541 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ed0801a506a0450ae1dfd79efd3da34f926275203f45c6b76d9baf31db1319" Oct 31 03:40:49.306511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.306485 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.306687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.306633 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.306790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.306773 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9dk\" (UniqueName: \"kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.307343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.307319 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.309063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.309040 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.368504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.368427 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9dk\" (UniqueName: \"kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk\") pod \"maas-api-revoke-sub-9cc73754-2hn4m\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.408159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.408082 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:49.792810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:49.792784 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m"] Oct 31 03:40:49.793423 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:49.793392 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692e81fc_bdd4_4a7d_82c4_093f3d61a84b.slice/crio-b6df054873e39218461ae51226d7558b4e015426f1df872a1eb105b5abcde494 WatchSource:0}: Error finding container b6df054873e39218461ae51226d7558b4e015426f1df872a1eb105b5abcde494: Status 404 returned error can't find the container with id b6df054873e39218461ae51226d7558b4e015426f1df872a1eb105b5abcde494 Oct 31 03:40:50.284656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:50.284619 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" event={"ID":"692e81fc-bdd4-4a7d-82c4-093f3d61a84b","Type":"ContainerStarted","Data":"ee7d500ef553a0f9d8d67c755b9694e4d3a5803acba13ef4000dc5c27bad900d"} Oct 31 03:40:50.285165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:50.284689 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" event={"ID":"692e81fc-bdd4-4a7d-82c4-093f3d61a84b","Type":"ContainerStarted","Data":"b6df054873e39218461ae51226d7558b4e015426f1df872a1eb105b5abcde494"} Oct 31 03:40:50.403170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:50.403049 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" podStartSLOduration=1.403033477 podStartE2EDuration="1.403033477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:50.399643303 +0000 UTC m=+1278.196361561" watchObservedRunningTime="2026-10-31 03:40:50.403033477 +0000 UTC m=+1278.199751734" Oct 31 03:40:51.293421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:51.293385 4488 generic.go:356] "Generic (PLEG): container finished" podID="692e81fc-bdd4-4a7d-82c4-093f3d61a84b" containerID="ee7d500ef553a0f9d8d67c755b9694e4d3a5803acba13ef4000dc5c27bad900d" exitCode=0 Oct 31 03:40:51.293868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:51.293474 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" event={"ID":"692e81fc-bdd4-4a7d-82c4-093f3d61a84b","Type":"ContainerDied","Data":"ee7d500ef553a0f9d8d67c755b9694e4d3a5803acba13ef4000dc5c27bad900d"} Oct 31 03:40:52.473535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.473500 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:52.652847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.652811 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk\" (UniqueName: \"kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk\") pod \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " Oct 31 03:40:52.653031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.652922 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca\") pod \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " Oct 31 03:40:52.653031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.652944 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca\") pod \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\" (UID: \"692e81fc-bdd4-4a7d-82c4-093f3d61a84b\") " Oct 31 03:40:52.653382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.653358 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca" pod "692e81fc-bdd4-4a7d-82c4-093f3d61a84b" (UID: "692e81fc-bdd4-4a7d-82c4-093f3d61a84b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:52.653643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.653624 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:52.655436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.655407 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk" pod "692e81fc-bdd4-4a7d-82c4-093f3d61a84b" (UID: "692e81fc-bdd4-4a7d-82c4-093f3d61a84b"). InnerVolumeSpecName "kube-api-access-hv9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:52.655537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.655514 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca" pod "692e81fc-bdd4-4a7d-82c4-093f3d61a84b" (UID: "692e81fc-bdd4-4a7d-82c4-093f3d61a84b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:52.755567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.755535 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9dk\" (UniqueName: \"kubernetes.io/projected/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-kube-api-access-hv9dk\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:52.755567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:52.755565 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692e81fc-bdd4-4a7d-82c4-093f3d61a84b-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.111518 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:53.111486 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:53.320595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:53.320555 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" Oct 31 03:40:53.320595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:53.320593 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m" event={"ID":"692e81fc-bdd4-4a7d-82c4-093f3d61a84b","Type":"ContainerDied","Data":"b6df054873e39218461ae51226d7558b4e015426f1df872a1eb105b5abcde494"} Oct 31 03:40:53.320839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:53.320640 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6df054873e39218461ae51226d7558b4e015426f1df872a1eb105b5abcde494" Oct 31 03:40:55.050511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.050476 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc"] Oct 31 03:40:55.100255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.100218 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e28563-n6lpc"] Oct 31 03:40:55.208160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.208123 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm"] Oct 31 03:40:55.208991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.208961 4488 cpu_manager.go:414] "containerMap: removing container" podUID="692e81fc-bdd4-4a7d-82c4-093f3d61a84b" containerName="revoke-keys" Oct 31 03:40:55.208991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.208992 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692e81fc-bdd4-4a7d-82c4-093f3d61a84b" containerName="revoke-keys" Oct 31 03:40:55.209269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.209253 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="692e81fc-bdd4-4a7d-82c4-093f3d61a84b" containerName="revoke-keys" Oct 31 03:40:55.212756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.212733 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.242044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.242010 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp"] Oct 31 03:40:55.246372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.246348 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.271541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.271506 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm"] Oct 31 03:40:55.287489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.287459 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:40:55.287489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.287488 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:40:55.287728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.287506 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:40:55.287728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.287557 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:40:55.289878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.289854 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.289931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.289885 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.289931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.289910 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwvh\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.290114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290099 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.290155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290144 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.290217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290205 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.290250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290241 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.290283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290260 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.290283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290276 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.290362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290344 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtr6\" (UniqueName: \"kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.290462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290446 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.290508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.290495 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.329101 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.329035 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp"] Oct 31 03:40:55.391876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.391828 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.391920 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392001 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392025 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392333 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwvh\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392467 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392392 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392471 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392479 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392544 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.392641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392597 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.392756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392660 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392695 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392720 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.392756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.392747 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtr6\" (UniqueName: \"kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.393193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.393155 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.393288 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.393233 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.393502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.393473 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.393888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.393800 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.395809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.395782 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.396235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.396214 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.396328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.396282 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.493173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.493140 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwvh\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.496404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.496374 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtr6\" (UniqueName: \"kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6\") pod \"maas-api-revoke-sub-0021d397-mjkzm\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.496577 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.496552 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.528055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.528019 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:40:55.561586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.561550 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:55.902903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.902824 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq"] Oct 31 03:40:55.908992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.908966 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:55.914830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.914801 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm"] Oct 31 03:40:55.953599 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:55.953571 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873d976a_5727_4493_87d1_17c3c92933ac.slice/crio-c0aec0454b85b47966a5d1b45f44e2eed1951afaaefba95dcdf318f2617916bc WatchSource:0}: Error finding container c0aec0454b85b47966a5d1b45f44e2eed1951afaaefba95dcdf318f2617916bc: Status 404 returned error can't find the container with id c0aec0454b85b47966a5d1b45f44e2eed1951afaaefba95dcdf318f2617916bc Oct 31 03:40:55.968699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.968666 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq"] Oct 31 03:40:55.992798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.992148 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:55.992798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.992235 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:55.992798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.992288 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:55.992983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:55.992956 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp"] Oct 31 03:40:56.001325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001290 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001363 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001495 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001566 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001633 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001684 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001749 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxq4\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001800 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.001867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.001821 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103398 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103464 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103511 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103565 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxq4\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103617 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103648 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103708 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103791 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.103931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103884 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.104418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.103892 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.104418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.104201 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.104418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.104300 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.104576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.104448 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.104897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.104875 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.106330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.106308 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.106727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.106707 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.165735 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.165641 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.165880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.165755 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxq4\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.226429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.226399 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:56.339426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.339388 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dd0af4-353e-4aa9-af17-43a22de131a9" path="/var/lib/kubelet/pods/06dd0af4-353e-4aa9-af17-43a22de131a9/volumes" Oct 31 03:40:56.339867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.339844 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv"] Oct 31 03:40:56.347172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.345392 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z"] Oct 31 03:40:56.347172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.345543 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.350565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.350542 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.386339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.385376 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv"] Oct 31 03:40:56.386527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.386409 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" event={"ID":"873d976a-5727-4493-87d1-17c3c92933ac","Type":"ContainerStarted","Data":"21ff372bac3a5a7ddf7dd1180eb7b02c88db14d9e5f9d747693eec7ed2832a86"} Oct 31 03:40:56.386527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.386463 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" event={"ID":"873d976a-5727-4493-87d1-17c3c92933ac","Type":"ContainerStarted","Data":"c0aec0454b85b47966a5d1b45f44e2eed1951afaaefba95dcdf318f2617916bc"} Oct 31 03:40:56.388785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.388760 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" event={"ID":"af6be095-5dfa-42f3-8cbe-fd404adcd57e","Type":"ContainerStarted","Data":"b3f99405a818b994ed7a5e384b3a975d9409e25c1eb2bb0c29df97fcf00ff840"} Oct 31 03:40:56.389049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.389026 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" event={"ID":"af6be095-5dfa-42f3-8cbe-fd404adcd57e","Type":"ContainerStarted","Data":"4b63ff284ac69bfd775f7d7f7dcdf3149c162e19d182621f6b6cb93475836caa"} Oct 31 03:40:56.419368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.419277 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.427189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.419595 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88dz\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.443542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.443509 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.443779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.443759 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.444008 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.443990 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.444176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.444157 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.444323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.444306 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxpp\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.445246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445219 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.445352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445320 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.445415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445393 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.445492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445473 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.445575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445559 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.445639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445627 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.445691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445661 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.445744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445712 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.445887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445868 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.445982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.445967 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.446044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.446015 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.454987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.454950 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z"] Oct 31 03:40:56.547424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547385 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.547424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547433 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.547686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547474 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.547686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547498 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88dz\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.547686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547532 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.547916 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547870 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.548073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547952 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.548073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547967 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.548073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.547889 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.548073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548006 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.548073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548039 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxpp\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548113 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548145 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548183 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548237 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548290 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548361 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548418 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548470 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548549 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548578 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548750 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548924 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.548190 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.549285 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.549963 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.550547 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.550947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.550660 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.551818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.551624 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.551887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.551872 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.552280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.552236 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.552847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.552822 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.562643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.562608 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:40:56.631836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.631473 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.632638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.632607 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq"] Oct 31 03:40:56.633930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.633460 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88dz\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.635684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.635661 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.636114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.635759 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxpp\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.640916 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:56.640877 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066b1561_4bf0_49fc_835e_990f92496c8e.slice/crio-bc23c1763dd3e7dae7c2d2829c7b1bdd4b0644c97a3c44f2765d07d7cdfb3904 WatchSource:0}: Error finding container bc23c1763dd3e7dae7c2d2829c7b1bdd4b0644c97a3c44f2765d07d7cdfb3904: Status 404 returned error can't find the container with id bc23c1763dd3e7dae7c2d2829c7b1bdd4b0644c97a3c44f2765d07d7cdfb3904 Oct 31 03:40:56.643757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.643712 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:56.643865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.643803 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:56.643865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.643846 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:56.669072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.669036 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:56.674202 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.673980 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:56.744662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:56.744148 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podStartSLOduration=1.744126633 podStartE2EDuration="1.744126633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:56.739567676 +0000 UTC m=+1284.536285934" watchObservedRunningTime="2026-10-31 03:40:56.744126633 +0000 UTC m=+1284.540844893" Oct 31 03:40:57.129117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.120135 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" podStartSLOduration=2.120112583 podStartE2EDuration="2.120112583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:56.870016719 +0000 UTC m=+1284.666734978" watchObservedRunningTime="2026-10-31 03:40:57.120112583 +0000 UTC m=+1284.916830845" Oct 31 03:40:57.136292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.135815 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z"] Oct 31 03:40:57.138443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.136909 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv"] Oct 31 03:40:57.140732 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:40:57.140618 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4effa134_50cc_4790_b168_e246266aeaf1.slice/crio-c3eb419f41a031f3129aa45ae5012f4dd45a5436dd6f347612b1cd0d3eb4bec5 WatchSource:0}: Error finding container c3eb419f41a031f3129aa45ae5012f4dd45a5436dd6f347612b1cd0d3eb4bec5: Status 404 returned error can't find the container with id c3eb419f41a031f3129aa45ae5012f4dd45a5436dd6f347612b1cd0d3eb4bec5 Oct 31 03:40:57.140983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.140949 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:57.141136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.141050 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:57.141136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.141115 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:57.143972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.143516 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:57.144157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.144130 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:57.144282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.144263 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:40:57.485114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.484393 4488 generic.go:356] "Generic (PLEG): container finished" podID="873d976a-5727-4493-87d1-17c3c92933ac" containerID="21ff372bac3a5a7ddf7dd1180eb7b02c88db14d9e5f9d747693eec7ed2832a86" exitCode=0 Oct 31 03:40:57.485114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.484649 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" event={"ID":"873d976a-5727-4493-87d1-17c3c92933ac","Type":"ContainerDied","Data":"21ff372bac3a5a7ddf7dd1180eb7b02c88db14d9e5f9d747693eec7ed2832a86"} Oct 31 03:40:57.497473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.497434 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" event={"ID":"066b1561-4bf0-49fc-835e-990f92496c8e","Type":"ContainerStarted","Data":"0747546007d747cef987b3f74770424a36bd625f31f055d053c9b4802d9f3019"} Oct 31 03:40:57.497654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.497624 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" event={"ID":"066b1561-4bf0-49fc-835e-990f92496c8e","Type":"ContainerStarted","Data":"bc23c1763dd3e7dae7c2d2829c7b1bdd4b0644c97a3c44f2765d07d7cdfb3904"} Oct 31 03:40:57.500986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.500503 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" event={"ID":"42c6c416-ff4e-4d46-a1d8-8d188260e9ee","Type":"ContainerStarted","Data":"6b2cb37cdbc97d8851ffd0de4ab0d297c2309e67b10b147372215acd99a7bb16"} Oct 31 03:40:57.500986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.500685 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" event={"ID":"42c6c416-ff4e-4d46-a1d8-8d188260e9ee","Type":"ContainerStarted","Data":"1d9e66aa5835342b8310ace09d22c96f604766973c1312059bf41d5837eafa1e"} Oct 31 03:40:57.503039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.503005 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" event={"ID":"4effa134-50cc-4790-b168-e246266aeaf1","Type":"ContainerStarted","Data":"2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95"} Oct 31 03:40:57.503154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.503059 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" event={"ID":"4effa134-50cc-4790-b168-e246266aeaf1","Type":"ContainerStarted","Data":"c3eb419f41a031f3129aa45ae5012f4dd45a5436dd6f347612b1cd0d3eb4bec5"} Oct 31 03:40:57.577965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.577924 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:40:57.578255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.578229 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:40:57.684343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.684248 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:40:57.684656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.684641 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:40:57.794117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.791800 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podStartSLOduration=2.791778796 podStartE2EDuration="2.791778796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.790408069 +0000 UTC m=+1285.587126331" watchObservedRunningTime="2026-10-31 03:40:57.791778796 +0000 UTC m=+1285.588497056" Oct 31 03:40:57.983282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:57.979986 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podStartSLOduration=1.979962992 podStartE2EDuration="1.979962992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.972573208 +0000 UTC m=+1285.769291493" watchObservedRunningTime="2026-10-31 03:40:57.979962992 +0000 UTC m=+1285.776681253" Oct 31 03:40:58.126117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.102006 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podStartSLOduration=2.101984223 podStartE2EDuration="2.101984223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:58.098445457 +0000 UTC m=+1285.895163718" watchObservedRunningTime="2026-10-31 03:40:58.101984223 +0000 UTC m=+1285.898702482" Oct 31 03:40:58.182277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.179317 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:58.238180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.237957 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:40:58.248115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.240179 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.248115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.240249 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:40:58.579770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.579736 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.603436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.603397 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:40:58.695641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.695604 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.697513 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.697470 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:40:58.697797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.697779 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:40:58.697937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:58.697918 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:40:59.226115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.224197 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:59.226115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.224889 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:40:59.252118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.227105 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.252118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.227153 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:40:59.428187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.427422 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:40:59.428187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.427880 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:59.653276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.592869 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:59.703217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.694703 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.703217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.694760 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:40:59.703217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.694842 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.703217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.694865 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:40:59.767148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.743243 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:40:59.767148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.757250 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:40:59.836016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.819959 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:40:59.946114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:40:59.928208 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:00.009927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.008646 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.009927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.009630 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:00.034839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.034544 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.035045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.034871 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.051601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.049591 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.064116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.059173 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.068314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.068272 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.092181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.075316 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:00.092181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.082186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.092181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.088304 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:41:00.238346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.238296 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.252163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.242375 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.253347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.253315 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:41:00.390322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.390289 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:41:00.416115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.321780 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.416115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.408555 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:41:00.416115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.327536 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6\" (UniqueName: \"kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6\") pod \"873d976a-5727-4493-87d1-17c3c92933ac\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " Oct 31 03:41:00.416115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.409048 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca\") pod \"873d976a-5727-4493-87d1-17c3c92933ac\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " Oct 31 03:41:00.416115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.409142 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca\") pod \"873d976a-5727-4493-87d1-17c3c92933ac\" (UID: \"873d976a-5727-4493-87d1-17c3c92933ac\") " Oct 31 03:41:00.454051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.454001 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca" pod "873d976a-5727-4493-87d1-17c3c92933ac" (UID: "873d976a-5727-4493-87d1-17c3c92933ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:00.454440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.454401 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:00.468237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.468185 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:41:00.656189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.616226 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.656189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.616295 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:41:00.749370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.716288 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.749370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.741848 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:00.751391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.749808 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.751391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.749885 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:00.814322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.813562 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:41:00.814322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.813647 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:41:00.814762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.814733 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6" pod "873d976a-5727-4493-87d1-17c3c92933ac" (UID: "873d976a-5727-4493-87d1-17c3c92933ac"). InnerVolumeSpecName "kube-api-access-zjtr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:00.816036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.815984 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca" pod "873d976a-5727-4493-87d1-17c3c92933ac" (UID: "873d976a-5727-4493-87d1-17c3c92933ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:00.824336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.824257 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:00.885174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.862287 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtr6\" (UniqueName: \"kubernetes.io/projected/873d976a-5727-4493-87d1-17c3c92933ac-kube-api-access-zjtr6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:00.885174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.862335 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873d976a-5727-4493-87d1-17c3c92933ac-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:00.998934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:00.998819 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:01.035614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.035562 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.035871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.035840 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.037356 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.037419 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.037457 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.037500 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.037533 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.037565 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.062495 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.062915 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.063280 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:41:01.066444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.063341 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:41:01.112117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.091280 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.112117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.091635 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.121369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.121322 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:01.149723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.149690 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:41:01.175374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.175344 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:41:01.175689 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.175670 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:41:01.183220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.163203 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.196126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.183726 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:41:01.232161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.231074 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:41:01.232161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.231158 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:41:01.260186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.247181 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.260846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.260785 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.461941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.461909 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6"] Oct 31 03:41:01.462546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.462231 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.502847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.502799 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.503992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.503958 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:41:01.504306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.504289 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" Oct 31 03:41:01.528984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.528892 4488 cpu_manager.go:414] "containerMap: removing container" podUID="873d976a-5727-4493-87d1-17c3c92933ac" containerName="revoke-keys" Oct 31 03:41:01.529276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.529257 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873d976a-5727-4493-87d1-17c3c92933ac" containerName="revoke-keys" Oct 31 03:41:01.551116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.529669 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="873d976a-5727-4493-87d1-17c3c92933ac" containerName="revoke-keys" Oct 31 03:41:01.568117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.557145 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm" event={"ID":"873d976a-5727-4493-87d1-17c3c92933ac","Type":"ContainerDied","Data":"c0aec0454b85b47966a5d1b45f44e2eed1951afaaefba95dcdf318f2617916bc"} Oct 31 03:41:01.593118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.583184 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0aec0454b85b47966a5d1b45f44e2eed1951afaaefba95dcdf318f2617916bc" Oct 31 03:41:01.593118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.568825 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:41:01.593118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.583272 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:41:01.593118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.558215 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.670117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.646750 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:01.670117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.646930 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:41:01.670117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.647004 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:41:01.670117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.647083 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:41:01.671031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.670908 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:01.671031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.670967 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:01.700143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.694268 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz57d\" (UniqueName: \"kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.700143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.694347 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.700143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.694380 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.708259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.707239 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:41:01.708259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.707300 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:41:01.807392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.804362 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz57d\" (UniqueName: \"kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.807392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.804451 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.807392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.804487 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.807392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.805617 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:01.830770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.830725 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:01.898340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:01.897253 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6"] Oct 31 03:41:02.045121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.039933 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:02.072176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.064252 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.072176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.064690 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.072176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.066523 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:02.072176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.066794 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.133119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.072638 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.133119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.073028 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:41:02.198116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.150167 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:02.199453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.152177 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.204501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.153887 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.314329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.191186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:02.342967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.199006 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz57d\" (UniqueName: \"kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d\") pod \"maas-api-revoke-sub-a77a243d-ph6t6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:02.343280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.242671 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 03:41:02.422118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.343631 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.422118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.344122 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.422118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.344163 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:41:02.422118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.368811 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:41:02.422118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.383784 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:02.423037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.423006 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:02.512744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.502245 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:02.693830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.547309 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.695473 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.695532 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.709014 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.709072 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.715692 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.716161 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:02.734120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.716201 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:02.865118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:02.833810 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.121199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.121117 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.131072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.131021 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.141976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.141932 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:03.143572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.143533 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.179182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.151875 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:03.193543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.186570 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:41:03.193543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.187071 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.210854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.206210 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.210854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.210374 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:03.324320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.324271 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:41:03.384405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.361787 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:03.384405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.363116 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.388835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.388794 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.431254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.420079 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.431254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.420171 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:41:03.509117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.505880 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.551847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.551807 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.614118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.583647 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.614118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.583706 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:41:03.701387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.699771 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.701387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.699829 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:03.727161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.724160 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:41:03.727161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.724221 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:41:03.992942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.992740 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:03.996265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:03.993392 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:41:04.165123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.147466 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:04.165123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.149362 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:04.181132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.165813 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.181132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.172218 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.192530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.192476 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:04.217138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.209793 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.217138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.211386 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.232165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.231799 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:41:04.232165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.231870 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:41:04.254199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.254043 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.396118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.351979 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.396118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.363338 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.396118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.363750 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:04.461358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.447336 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.539732 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:04.500293 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394815d9_aa67_4f97_a741_e875f54fbbd6.slice/crio-86ec747b8901a311c7b05a7d02fc19e32bcd9de4ce99d35c62880b6585e6470a WatchSource:0}: Error finding container 86ec747b8901a311c7b05a7d02fc19e32bcd9de4ce99d35c62880b6585e6470a: Status 404 returned error can't find the container with id 86ec747b8901a311c7b05a7d02fc19e32bcd9de4ce99d35c62880b6585e6470a Oct 31 03:41:04.539732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.535211 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:04.584622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.584575 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:04.587730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.587700 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 03:41:04.587985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.587930 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 03:41:04.689544 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.689494 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:41:04.689799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.689763 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:41:04.690273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.690252 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:04.693234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.693200 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:04.779229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.777815 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6"] Oct 31 03:41:04.779229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:04.778667 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:05.009885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.009820 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.011263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.011216 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:41:05.156113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.150175 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.156113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.150590 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:41:05.156113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.152510 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:05.161448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.160555 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.164058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.163191 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" event={"ID":"394815d9-aa67-4f97-a741-e875f54fbbd6","Type":"ContainerStarted","Data":"86ec747b8901a311c7b05a7d02fc19e32bcd9de4ce99d35c62880b6585e6470a"} Oct 31 03:41:05.166759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.165999 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.208618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.206318 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:41:05.221338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.209204 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:05.225762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.223357 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:41:05.227910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.227778 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 03:41:05.227910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.227832 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 03:41:05.248844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.245882 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:41:05.284588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.283521 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:41:05.285300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.285263 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.340512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.340471 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:41:05.466397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.466368 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:41:05.538336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.538301 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.708611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.672217 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:05.708611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.673812 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:05.708611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.673852 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:05.708611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.686590 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:41:05.708611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.686646 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 03:41:05.721957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.719160 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:41:05.721957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.720589 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:41:05.723011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:05.722933 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:41:06.027119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.026810 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:41:06.046557 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.039535 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:41:06.058896 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.058864 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:41:06.063394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.063366 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:41:06.064557 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.059214 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:06.086995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.086970 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:41:06.121729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.121688 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:41:06.162143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.161785 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:06.200954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.200912 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj"] Oct 31 03:41:06.287673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.255996 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:41:06.287673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.278581 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8439c8dc-jszkj"] Oct 31 03:41:06.290137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.288635 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:41:06.340063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.339925 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1132c8-7514-48bd-bd3e-c266c5f36ad3" path="/var/lib/kubelet/pods/9b1132c8-7514-48bd-bd3e-c266c5f36ad3/volumes" Oct 31 03:41:06.342219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.342194 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:41:06.343294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.343116 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:41:06.343294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.343198 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" event={"ID":"394815d9-aa67-4f97-a741-e875f54fbbd6","Type":"ContainerStarted","Data":"47691a27f673e21d2416c3e59bc4fd38dbda580f627e1cf3a31f3c85f9acd9b6"} Oct 31 03:41:06.357120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.356878 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd"] Oct 31 03:41:06.362447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.362418 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.445362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.445308 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.460320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.460137 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dk4n\" (UniqueName: \"kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.470450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.446313 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd"] Oct 31 03:41:06.471105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.470712 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.573509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.573466 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.573896 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.573869 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.585391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.585359 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.586033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.585606 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dk4n\" (UniqueName: \"kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.612027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.611828 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.670305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.670176 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 03:41:06.670305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.670234 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 03:41:06.692526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.684791 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dk4n\" (UniqueName: \"kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n\") pod \"maas-api-revoke-sub-8702ed28-z6fnd\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:06.699661 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:41:06.699588 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:41:06.715554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:06.715519 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:07.296519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.293881 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:41:07.316270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.308066 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:41:07.337216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.332818 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:41:07.413197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.411429 4488 generic.go:356] "Generic (PLEG): container finished" podID="394815d9-aa67-4f97-a741-e875f54fbbd6" containerID="47691a27f673e21d2416c3e59bc4fd38dbda580f627e1cf3a31f3c85f9acd9b6" exitCode=0 Oct 31 03:41:07.413197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.412737 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" event={"ID":"394815d9-aa67-4f97-a741-e875f54fbbd6","Type":"ContainerDied","Data":"47691a27f673e21d2416c3e59bc4fd38dbda580f627e1cf3a31f3c85f9acd9b6"} Oct 31 03:41:07.712594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.708217 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd"] Oct 31 03:41:07.712594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.711035 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:41:07.713001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.712822 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:41:07.714351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.714329 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:41:07.769336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:07.769250 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" podStartSLOduration=6.76923255 podStartE2EDuration="6.76923255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:07.769147675 +0000 UTC m=+1295.565865933" watchObservedRunningTime="2026-10-31 03:41:07.76923255 +0000 UTC m=+1295.565950807" Oct 31 03:41:08.240120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.240000 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh"] Oct 31 03:41:08.249590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.249564 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.301283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.301198 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh"] Oct 31 03:41:08.370364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.370151 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nxx\" (UniqueName: \"kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.378001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.377965 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.378205 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.378111 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.434333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.434298 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" event={"ID":"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562","Type":"ContainerStarted","Data":"14cd78d81496b2824aff6b1c653e96f70a6e3e2cd779e2e2bb6db6fd192879ed"} Oct 31 03:41:08.437631 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.437598 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" event={"ID":"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562","Type":"ContainerStarted","Data":"b2fdd121014c57ac0cca6cd90571e26b01188d2a587fbfe7cd11d227dcf22d05"} Oct 31 03:41:08.488440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.479767 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nxx\" (UniqueName: \"kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.488440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.479966 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.488440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.480151 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.488440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.481215 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.504188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.500755 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.565200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.560295 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" podStartSLOduration=2.560272303 podStartE2EDuration="2.560272303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:08.559377139 +0000 UTC m=+1296.356095410" watchObservedRunningTime="2026-10-31 03:41:08.560272303 +0000 UTC m=+1296.356990562" Oct 31 03:41:08.565985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.565956 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nxx\" (UniqueName: \"kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx\") pod \"maas-api-revoke-sub-837b6d4c-tlbnh\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.604072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.604038 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:08.718540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.718048 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:08.886997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.886965 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca\") pod \"394815d9-aa67-4f97-a741-e875f54fbbd6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " Oct 31 03:41:08.887476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.887449 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d\" (UniqueName: \"kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d\") pod \"394815d9-aa67-4f97-a741-e875f54fbbd6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " Oct 31 03:41:08.887618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.887607 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca\") pod \"394815d9-aa67-4f97-a741-e875f54fbbd6\" (UID: \"394815d9-aa67-4f97-a741-e875f54fbbd6\") " Oct 31 03:41:08.888496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.888466 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca" pod "394815d9-aa67-4f97-a741-e875f54fbbd6" (UID: "394815d9-aa67-4f97-a741-e875f54fbbd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:08.891910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.891877 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca" pod "394815d9-aa67-4f97-a741-e875f54fbbd6" (UID: "394815d9-aa67-4f97-a741-e875f54fbbd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:08.895491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.895473 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d" pod "394815d9-aa67-4f97-a741-e875f54fbbd6" (UID: "394815d9-aa67-4f97-a741-e875f54fbbd6"). InnerVolumeSpecName "kube-api-access-wz57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:08.989320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.989284 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:08.989320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.989316 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz57d\" (UniqueName: \"kubernetes.io/projected/394815d9-aa67-4f97-a741-e875f54fbbd6-kube-api-access-wz57d\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:08.989514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:08.989330 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394815d9-aa67-4f97-a741-e875f54fbbd6-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:09.090345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.072683 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:41:09.115965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.115913 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh"] Oct 31 03:41:09.116692 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:09.116662 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e68a3ba_5351_4125_b422_c4f0dc042288.slice/crio-cedd5fe4f94b5a7897b55d0257390b40b6eea6864018d7afcbb4ed43331e27ff WatchSource:0}: Error finding container cedd5fe4f94b5a7897b55d0257390b40b6eea6864018d7afcbb4ed43331e27ff: Status 404 returned error can't find the container with id cedd5fe4f94b5a7897b55d0257390b40b6eea6864018d7afcbb4ed43331e27ff Oct 31 03:41:09.444803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.444709 4488 generic.go:356] "Generic (PLEG): container finished" podID="3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" containerID="14cd78d81496b2824aff6b1c653e96f70a6e3e2cd779e2e2bb6db6fd192879ed" exitCode=0 Oct 31 03:41:09.444803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.444790 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" event={"ID":"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562","Type":"ContainerDied","Data":"14cd78d81496b2824aff6b1c653e96f70a6e3e2cd779e2e2bb6db6fd192879ed"} Oct 31 03:41:09.446476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.446446 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" Oct 31 03:41:09.446476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.446462 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6" event={"ID":"394815d9-aa67-4f97-a741-e875f54fbbd6","Type":"ContainerDied","Data":"86ec747b8901a311c7b05a7d02fc19e32bcd9de4ce99d35c62880b6585e6470a"} Oct 31 03:41:09.446692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.446503 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ec747b8901a311c7b05a7d02fc19e32bcd9de4ce99d35c62880b6585e6470a" Oct 31 03:41:09.448147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:09.448080 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" event={"ID":"4e68a3ba-5351-4125-b422-c4f0dc042288","Type":"ContainerStarted","Data":"cedd5fe4f94b5a7897b55d0257390b40b6eea6864018d7afcbb4ed43331e27ff"} Oct 31 03:41:10.073480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:10.072863 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:10.941117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:10.940055 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8"] Oct 31 03:41:10.965118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:10.941640 4488 cpu_manager.go:414] "containerMap: removing container" podUID="394815d9-aa67-4f97-a741-e875f54fbbd6" containerName="revoke-keys" Oct 31 03:41:10.965118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:10.941666 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394815d9-aa67-4f97-a741-e875f54fbbd6" containerName="revoke-keys" Oct 31 03:41:10.965118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:10.942001 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="394815d9-aa67-4f97-a741-e875f54fbbd6" containerName="revoke-keys" Oct 31 03:41:11.011309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.002483 4488 generic.go:356] "Generic (PLEG): container finished" podID="4e68a3ba-5351-4125-b422-c4f0dc042288" containerID="74b8432956d7c0ae6c016b4db036f450b0b2cdafb210fbc91b9082f260ddeefc" exitCode=0 Oct 31 03:41:11.011309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.002992 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" event={"ID":"4e68a3ba-5351-4125-b422-c4f0dc042288","Type":"ContainerDied","Data":"74b8432956d7c0ae6c016b4db036f450b0b2cdafb210fbc91b9082f260ddeefc"} Oct 31 03:41:11.016545 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.016513 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.019881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.016882 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8"] Oct 31 03:41:11.046218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.042615 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.056123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.046510 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.056439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.056404 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.056647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.056628 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.056920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.056901 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7vd\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.057217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.057196 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.057375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.057358 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.057514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.057500 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.087117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.070448 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.089776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.068265 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:11.165282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.165229 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:11.165911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.165881 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214432 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214502 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214538 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214580 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214618 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214725 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7vd\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214815 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214847 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.214891 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.215323 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.218164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.215886 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.244118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.222905 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.267232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.267203 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.298123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.277226 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:11.321764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.306273 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.323763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.323712 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.367109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.366795 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.378866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.378826 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7vd\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.417141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.411722 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.417141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.412140 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.462113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.457304 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.490170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.473570 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.490170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.485196 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:11.490170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.489465 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:11.851298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:11.845336 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.289761 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.290317 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.291356 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.292115 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.292441 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.298114 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.298471 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.298744 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.301916 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.302003 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.302817 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.302879 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:41:12.335540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.325359 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 03:41:12.346116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.338011 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.375746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.375268 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.470136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.459441 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.470136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.460137 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.493130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.490234 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.493130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.490703 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:12.618212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.548358 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:12.765636 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.764239 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": context deadline exceeded" Oct 31 03:41:12.940117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.927853 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:41:12.944265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.944228 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:41:12.944531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.944517 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:41:12.944675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.944663 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:41:12.960936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:12.947641 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.123115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.119242 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:41:13.299265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.299167 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:13.323367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.323321 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:13.324392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.324358 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:13.325638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.325617 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:41:13.350232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.350186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.393273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.393216 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:13.401159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.396736 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:13.401159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.396799 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:13.401813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.401780 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.419799 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.421900 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.422189 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.422425 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.422484 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.451044 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" event={"ID":"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562","Type":"ContainerDied","Data":"b2fdd121014c57ac0cca6cd90571e26b01188d2a587fbfe7cd11d227dcf22d05"} Oct 31 03:41:13.456118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.451133 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2fdd121014c57ac0cca6cd90571e26b01188d2a587fbfe7cd11d227dcf22d05" Oct 31 03:41:13.464887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.464845 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.466080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.466056 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:41:13.470552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.469268 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:13.520115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.516941 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.520115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.517277 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:13.537409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.537368 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n\" (UniqueName: \"kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n\") pod \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " Oct 31 03:41:13.549511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.549407 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca\") pod \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " Oct 31 03:41:13.555376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.555328 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca\") pod \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\" (UID: \"3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562\") " Oct 31 03:41:13.576047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.570326 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca" pod "3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" (UID: "3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:13.576047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.570968 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:13.576047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.571338 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:41:13.700232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.648706 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:13.700232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.655449 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca" pod "3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" (UID: "3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:13.803611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.803530 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:13.856055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.856010 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n" pod "3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" (UID: "3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562"). InnerVolumeSpecName "kube-api-access-6dk4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:13.994614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:13.994572 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.161920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.160572 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.255734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.255523 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dk4n\" (UniqueName: \"kubernetes.io/projected/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562-kube-api-access-6dk4n\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:14.311351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.293303 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:14.311351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.293397 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:14.311351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.293445 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.314686 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.394394 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.396236 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.396948 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.408189 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.408545 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.426399 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.426814 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:14.440173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.434491 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.467270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.467201 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.472503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.472465 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.523609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.479201 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:14.523609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.504302 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.523609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.522465 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:14.523609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.522925 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:14.625383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.603869 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:41:14.856119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:14.841376 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.228213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.196920 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.327394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.327339 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.461863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.461802 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:15.549057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.548984 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:15.549801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.549752 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:41:15.600490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.569320 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:15.624258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.569380 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:15.638152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.569420 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:15.638763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.569465 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.691819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.569543 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:15.702572 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:41:15.638214 4488 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.326s" Oct 31 03:41:15.728374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.728341 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8"] Oct 31 03:41:15.728697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.655345 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.847181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.793016 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:15.847181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.814139 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:15.847181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.814528 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:15.847181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:15.815071 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:16.137206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.102689 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" event={"ID":"0438525a-cdfc-4df1-9f73-701d38383aad","Type":"ContainerStarted","Data":"08856f05b434878a1d7eb57b48133192fdea4d01bad7cc123e5d3f4477f83c49"} Oct 31 03:41:16.137206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.103004 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd" Oct 31 03:41:16.220176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.220134 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.548347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.541245 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:16.597319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.581289 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.597319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.587364 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.683018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.682970 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:16.803607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.803498 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.804048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.804016 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:16.820522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.820480 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:16.822518 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821040 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.822797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821117 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:16.823035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821154 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:41:16.823236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821202 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:16.823406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821276 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:16.823581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821433 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.823753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821470 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:16.823915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.821536 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:16.885738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:16.885679 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded" Oct 31 03:41:17.185344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.149170 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.207946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.207865 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.222507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.222451 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.349422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.349392 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:17.519172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.511912 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:41:17.520308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.520274 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca\") pod \"4e68a3ba-5351-4125-b422-c4f0dc042288\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " Oct 31 03:41:17.522047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.520554 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx\" (UniqueName: \"kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx\") pod \"4e68a3ba-5351-4125-b422-c4f0dc042288\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " Oct 31 03:41:17.527506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.527467 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca\") pod \"4e68a3ba-5351-4125-b422-c4f0dc042288\" (UID: \"4e68a3ba-5351-4125-b422-c4f0dc042288\") " Oct 31 03:41:17.549166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.549115 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca" pod "4e68a3ba-5351-4125-b422-c4f0dc042288" (UID: "4e68a3ba-5351-4125-b422-c4f0dc042288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:17.579116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.555021 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.740119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.630602 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:17.808326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.673484 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.673523 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.723466 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx" pod "4e68a3ba-5351-4125-b422-c4f0dc042288" (UID: "4e68a3ba-5351-4125-b422-c4f0dc042288"). InnerVolumeSpecName "kube-api-access-x8nxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.742586 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.825944 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.825983 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.826017 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.828232 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.829562 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.830133 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.830598 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.831484 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.834117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.832183 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.867040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.867010 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:41:17.944856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.899082 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:41:17.944856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.900405 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:41:17.944856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.903345 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:17.944856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.927183 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nxx\" (UniqueName: \"kubernetes.io/projected/4e68a3ba-5351-4125-b422-c4f0dc042288-kube-api-access-x8nxx\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:17.966909 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.966827 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" Oct 31 03:41:17.967614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.967527 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh" event={"ID":"4e68a3ba-5351-4125-b422-c4f0dc042288","Type":"ContainerDied","Data":"cedd5fe4f94b5a7897b55d0257390b40b6eea6864018d7afcbb4ed43331e27ff"} Oct 31 03:41:17.967614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.967590 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cedd5fe4f94b5a7897b55d0257390b40b6eea6864018d7afcbb4ed43331e27ff" Oct 31 03:41:17.969864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:17.968891 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:41:18.007624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.007589 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca" pod "4e68a3ba-5351-4125-b422-c4f0dc042288" (UID: "4e68a3ba-5351-4125-b422-c4f0dc042288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:18.037259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.028430 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:18.069574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.069464 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:41:18.087537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.087504 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:41:18.091409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.091382 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:41:18.112118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.109724 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e68a3ba-5351-4125-b422-c4f0dc042288-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:18.209398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.209369 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:41:18.214797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.214766 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:41:18.230055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:18.230027 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:41:19.044310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.044267 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" event={"ID":"0438525a-cdfc-4df1-9f73-701d38383aad","Type":"ContainerStarted","Data":"cfdd3b7bccd1fa4bf9c4a8125697d305d5f68a661f3198f7d73251659ecbfefc"} Oct 31 03:41:19.490898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.490863 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:19.492074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.492045 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:41:19.492246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.492115 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:41:19.540960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.540928 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5"] Oct 31 03:41:19.541525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.541507 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" containerName="revoke-keys" Oct 31 03:41:19.541525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.541523 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" containerName="revoke-keys" Oct 31 03:41:19.541662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.541565 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4e68a3ba-5351-4125-b422-c4f0dc042288" containerName="revoke-keys" Oct 31 03:41:19.541662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.541570 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e68a3ba-5351-4125-b422-c4f0dc042288" containerName="revoke-keys" Oct 31 03:41:19.541728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.541684 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" containerName="revoke-keys" Oct 31 03:41:19.541728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.541695 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e68a3ba-5351-4125-b422-c4f0dc042288" containerName="revoke-keys" Oct 31 03:41:19.546954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.546932 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.580857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.580281 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsfl\" (UniqueName: \"kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.580857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.580382 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.580857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.580739 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.599978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.599676 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:19.599978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.599719 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:41:19.599978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.599727 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:41:19.600834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.599995 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:41:19.636160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.627404 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5"] Oct 31 03:41:19.641187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.638142 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podStartSLOduration=9.638121365 podStartE2EDuration="9.638121365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:19.600210849 +0000 UTC m=+1307.396929110" watchObservedRunningTime="2026-10-31 03:41:19.638121365 +0000 UTC m=+1307.434839624" Oct 31 03:41:19.681789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.681754 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsfl\" (UniqueName: \"kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.681789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.681794 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.682039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.681999 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.682507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.682485 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.684587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.684559 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.743757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.743673 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsfl\" (UniqueName: \"kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl\") pod \"maas-api-revoke-sub-4e1ea513-lnzd5\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:19.863638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:19.863603 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:20.299030 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:20.299001 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf01b115_5196_42c5_950b_4dff7db06a0e.slice/crio-e26789a103fd5c675a01db6e73d0d64748ec07511408976ccfa61585623f3d41 WatchSource:0}: Error finding container e26789a103fd5c675a01db6e73d0d64748ec07511408976ccfa61585623f3d41: Status 404 returned error can't find the container with id e26789a103fd5c675a01db6e73d0d64748ec07511408976ccfa61585623f3d41 Oct 31 03:41:20.299787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:20.299763 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5"] Oct 31 03:41:20.491417 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:20.491382 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:41:20.491585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:20.491453 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:41:20.922128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:20.921235 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:21.065940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.065849 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" event={"ID":"cf01b115-5196-42c5-950b-4dff7db06a0e","Type":"ContainerStarted","Data":"dc1ddf85efa0ddcc8e13d73142268ee2d611198a3f1dffd4533f7302698fae41"} Oct 31 03:41:21.065940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.065902 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" event={"ID":"cf01b115-5196-42c5-950b-4dff7db06a0e","Type":"ContainerStarted","Data":"e26789a103fd5c675a01db6e73d0d64748ec07511408976ccfa61585623f3d41"} Oct 31 03:41:21.201751 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.201689 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" podStartSLOduration=2.201674958 podStartE2EDuration="2.201674958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:21.197223867 +0000 UTC m=+1308.993942125" watchObservedRunningTime="2026-10-31 03:41:21.201674958 +0000 UTC m=+1308.998393216" Oct 31 03:41:21.330686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.330656 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:21.443110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.434645 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km"] Oct 31 03:41:21.458498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.452435 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.492737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.491456 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:41:21.492737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.491516 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:41:21.508568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.508515 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dafd9df5-openshift-default-dockercfg-5fljk\"" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.556550 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km"] Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689434 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689579 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689617 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689645 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689685 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689719 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689923 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.689965 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.690016 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t6g\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.733214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.691104 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:21.792332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.792064 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.792332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.792146 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.792332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.792177 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.792332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.792214 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.792332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.792248 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.794789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.792424 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.795004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.794971 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.795131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.795043 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t6g\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.795204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.795184 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.796011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.795975 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.796145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.794215 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.796729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.796694 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.798543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.797039 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.798719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.797305 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.798824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.797612 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.844312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.843721 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.859603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.859532 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.864398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.864368 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t6g\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g\") pod \"e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:21.872790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:21.872756 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:22.082165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.079003 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:22.085399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.085362 4488 generic.go:356] "Generic (PLEG): container finished" podID="cf01b115-5196-42c5-950b-4dff7db06a0e" containerID="dc1ddf85efa0ddcc8e13d73142268ee2d611198a3f1dffd4533f7302698fae41" exitCode=0 Oct 31 03:41:22.085812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.085788 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" event={"ID":"cf01b115-5196-42c5-950b-4dff7db06a0e","Type":"ContainerDied","Data":"dc1ddf85efa0ddcc8e13d73142268ee2d611198a3f1dffd4533f7302698fae41"} Oct 31 03:41:22.088250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.088230 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:22.491368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.491263 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:41:22.491368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.491353 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:41:22.639380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.639351 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km"] Oct 31 03:41:22.642583 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:22.642035 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3722cbb_788e_403c_8e29_2430eb5d804b.slice/crio-55f3ca18db5cd28776d556875e623a73e14aa1484e45e35a7624568c8802e91e WatchSource:0}: Error finding container 55f3ca18db5cd28776d556875e623a73e14aa1484e45e35a7624568c8802e91e: Status 404 returned error can't find the container with id 55f3ca18db5cd28776d556875e623a73e14aa1484e45e35a7624568c8802e91e Oct 31 03:41:22.645256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.644987 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:22.645256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.645103 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:22.645256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:22.645164 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:23.073223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.073134 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:23.097921 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.097876 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" event={"ID":"d3722cbb-788e-403c-8e29-2430eb5d804b","Type":"ContainerStarted","Data":"1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297"} Oct 31 03:41:23.098126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.097949 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" event={"ID":"d3722cbb-788e-403c-8e29-2430eb5d804b","Type":"ContainerStarted","Data":"55f3ca18db5cd28776d556875e623a73e14aa1484e45e35a7624568c8802e91e"} Oct 31 03:41:23.339367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.339336 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:23.513116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.487585 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podStartSLOduration=2.487558619 podStartE2EDuration="2.487558619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:23.237038368 +0000 UTC m=+1311.033756627" watchObservedRunningTime="2026-10-31 03:41:23.487558619 +0000 UTC m=+1311.284276877" Oct 31 03:41:23.513116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.501137 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:41:23.513116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.501201 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:41:23.537395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.535624 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca\") pod \"cf01b115-5196-42c5-950b-4dff7db06a0e\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " Oct 31 03:41:23.537395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.535859 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca\") pod \"cf01b115-5196-42c5-950b-4dff7db06a0e\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " Oct 31 03:41:23.537395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.536173 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl\" (UniqueName: \"kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl\") pod \"cf01b115-5196-42c5-950b-4dff7db06a0e\" (UID: \"cf01b115-5196-42c5-950b-4dff7db06a0e\") " Oct 31 03:41:23.548560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.543432 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca" pod "cf01b115-5196-42c5-950b-4dff7db06a0e" (UID: "cf01b115-5196-42c5-950b-4dff7db06a0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:23.548560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.545769 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca" pod "cf01b115-5196-42c5-950b-4dff7db06a0e" (UID: "cf01b115-5196-42c5-950b-4dff7db06a0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:23.554556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.554455 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl" pod "cf01b115-5196-42c5-950b-4dff7db06a0e" (UID: "cf01b115-5196-42c5-950b-4dff7db06a0e"). InnerVolumeSpecName "kube-api-access-jcsfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:23.555559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.555522 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8"] Oct 31 03:41:23.556598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.556573 4488 cpu_manager.go:414] "containerMap: removing container" podUID="cf01b115-5196-42c5-950b-4dff7db06a0e" containerName="revoke-keys" Oct 31 03:41:23.556598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.556597 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf01b115-5196-42c5-950b-4dff7db06a0e" containerName="revoke-keys" Oct 31 03:41:23.556864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.556838 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf01b115-5196-42c5-950b-4dff7db06a0e" containerName="revoke-keys" Oct 31 03:41:23.562506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.562483 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.660115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.645857 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9e38c94c-openshift-default-dockercfg-vbmx5\"" Oct 31 03:41:23.660359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.653943 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.660546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.660528 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.660757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.660734 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkdx\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.660900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.660885 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.661049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.661035 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.661277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.661261 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.661412 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.661397 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.661659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.661638 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.661901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.661883 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.662184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.662166 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcsfl\" (UniqueName: \"kubernetes.io/projected/cf01b115-5196-42c5-950b-4dff7db06a0e-kube-api-access-jcsfl\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:23.662304 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.662288 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:23.662389 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.662378 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf01b115-5196-42c5-950b-4dff7db06a0e-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:23.679131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.678579 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8"] Oct 31 03:41:23.689044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.688837 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.767647 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768271 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768354 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768481 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768587 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768623 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768647 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768740 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkdx\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768768 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.768835 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.770433 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.771324 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.771529 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.779433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.772912 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.783071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.780990 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.783071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.783045 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.839575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.839536 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.844651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.844611 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkdx\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx\") pod \"e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:23.850312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.850285 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:23.986732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:23.986622 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:24.025797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.016553 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:24.093055 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.092430 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:24.124119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.118007 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" Oct 31 03:41:24.124119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.121799 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5" event={"ID":"cf01b115-5196-42c5-950b-4dff7db06a0e","Type":"ContainerDied","Data":"e26789a103fd5c675a01db6e73d0d64748ec07511408976ccfa61585623f3d41"} Oct 31 03:41:24.124119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.121867 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e26789a103fd5c675a01db6e73d0d64748ec07511408976ccfa61585623f3d41" Oct 31 03:41:24.124119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.121944 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:24.461597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.461556 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8"] Oct 31 03:41:24.461777 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:24.461749 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb7b302_9057_446b_abf7_90c8071fb932.slice/crio-479d9bda8fa75a9dba267dda1c3920950b94480eda86a7b78191b6be90567eee WatchSource:0}: Error finding container 479d9bda8fa75a9dba267dda1c3920950b94480eda86a7b78191b6be90567eee: Status 404 returned error can't find the container with id 479d9bda8fa75a9dba267dda1c3920950b94480eda86a7b78191b6be90567eee Oct 31 03:41:24.464806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.464762 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:24.464888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.464868 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:24.464940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.464925 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:24.491063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.491031 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:41:24.491215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:24.491125 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:41:25.129020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.128980 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" event={"ID":"8eb7b302-9057-446b-abf7-90c8071fb932","Type":"ContainerStarted","Data":"dd2a444b5219548823a821497f5d2ebeb1583b3001b0e8080e16691f8115f33f"} Oct 31 03:41:25.129572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.129055 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" event={"ID":"8eb7b302-9057-446b-abf7-90c8071fb932","Type":"ContainerStarted","Data":"479d9bda8fa75a9dba267dda1c3920950b94480eda86a7b78191b6be90567eee"} Oct 31 03:41:25.129572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.129494 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:25.130753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.130735 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:41:25.249965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.249909 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podStartSLOduration=2.249894145 podStartE2EDuration="2.249894145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:25.249365327 +0000 UTC m=+1313.046083585" watchObservedRunningTime="2026-10-31 03:41:25.249894145 +0000 UTC m=+1313.046612402" Oct 31 03:41:25.457214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.456571 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:25.587271 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.587233 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:25.587770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.587747 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:25.588778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.588759 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:41:25.615180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.615149 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:25.775645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.775562 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:25.986912 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.986867 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:25.992943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:25.992917 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:26.140227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:26.140195 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:26.140936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:26.140914 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:41:26.506063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:26.505984 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:27.125579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.125541 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w"] Oct 31 03:41:27.130180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.130158 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.216855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.216822 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84"] Oct 31 03:41:27.220991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.220971 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.237503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.237476 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w"] Oct 31 03:41:27.241227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.241184 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqz2\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.241390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.241286 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.241503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.241475 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.241900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.241798 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.241900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.241850 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.242150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.242132 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.242236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.242173 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.242236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.242200 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.242236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.242222 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.278351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.278314 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:27.278541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.278428 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:41:27.279013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.278879 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:41:27.282445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.282394 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:41:27.286168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.286140 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84"] Oct 31 03:41:27.344211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344179 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344223 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344310 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344358 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfdj\" (UniqueName: \"kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.344630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344599 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.344775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344662 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqz2\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344710 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344740 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344793 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344818 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.344910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.344873 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.345231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.345207 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.345332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.345246 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.345332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.345262 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.345471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.345385 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.345607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.345584 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.345762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.345737 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.346913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.346888 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.347443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.347425 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.408230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.408157 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.408230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.408172 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqz2\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.446684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.446645 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:27.447294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.447268 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.447521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.447500 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.447628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.447610 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfdj\" (UniqueName: \"kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.448198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.448174 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.450040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.450015 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.515855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.515824 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfdj\" (UniqueName: \"kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj\") pod \"maas-api-revoke-sub-7433c1c9-58f84\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.543359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.543324 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:27.887613 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:27.887579 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b225622_1fce_4260_981d_cc845888d525.slice/crio-bb0a757e48aefeb40423ab5fa62fd9b8724203717afaaf82851410ac43a69695 WatchSource:0}: Error finding container bb0a757e48aefeb40423ab5fa62fd9b8724203717afaaf82851410ac43a69695: Status 404 returned error can't find the container with id bb0a757e48aefeb40423ab5fa62fd9b8724203717afaaf82851410ac43a69695 Oct 31 03:41:27.888126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.888082 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w"] Oct 31 03:41:27.890819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.890770 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:27.890912 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.890856 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:27.890912 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.890889 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:27.947806 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:27.947767 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda112b709_d618_4b06_9592_1b9983b56daa.slice/crio-6a89db91c569b9e32524bc43ef2189681632ebf029e32179fd27de2617ed6ae9 WatchSource:0}: Error finding container 6a89db91c569b9e32524bc43ef2189681632ebf029e32179fd27de2617ed6ae9: Status 404 returned error can't find the container with id 6a89db91c569b9e32524bc43ef2189681632ebf029e32179fd27de2617ed6ae9 Oct 31 03:41:27.948079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:27.948052 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84"] Oct 31 03:41:28.162629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.162517 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" event={"ID":"6b225622-1fce-4260-981d-cc845888d525","Type":"ContainerStarted","Data":"eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9"} Oct 31 03:41:28.162629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.162587 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" event={"ID":"6b225622-1fce-4260-981d-cc845888d525","Type":"ContainerStarted","Data":"bb0a757e48aefeb40423ab5fa62fd9b8724203717afaaf82851410ac43a69695"} Oct 31 03:41:28.164250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.164219 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" event={"ID":"a112b709-d618-4b06-9592-1b9983b56daa","Type":"ContainerStarted","Data":"6a89db91c569b9e32524bc43ef2189681632ebf029e32179fd27de2617ed6ae9"} Oct 31 03:41:28.182956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.182920 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:28.287621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.287569 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podStartSLOduration=1.287554029 podStartE2EDuration="1.287554029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:28.283127916 +0000 UTC m=+1316.079846176" watchObservedRunningTime="2026-10-31 03:41:28.287554029 +0000 UTC m=+1316.084272286" Oct 31 03:41:28.378210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.378184 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:28.447681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.447592 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:28.451177 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.451142 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:28.451349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.451212 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:28.554471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.554441 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:28.999352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:28.999308 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:29.080726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.080693 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g"] Oct 31 03:41:29.085122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.085067 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.146349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.146307 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g"] Oct 31 03:41:29.153832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.153797 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:29.176309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.176276 4488 generic.go:356] "Generic (PLEG): container finished" podID="a112b709-d618-4b06-9592-1b9983b56daa" containerID="e9993bd242b0dc51ad087e233a06de3a06b797a2c69f4ac808452cdc1ab269e6" exitCode=0 Oct 31 03:41:29.176529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.176362 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" event={"ID":"a112b709-d618-4b06-9592-1b9983b56daa","Type":"ContainerDied","Data":"e9993bd242b0dc51ad087e233a06de3a06b797a2c69f4ac808452cdc1ab269e6"} Oct 31 03:41:29.177307 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.177285 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.177468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.177450 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srnq\" (UniqueName: \"kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.177541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.177519 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.279320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.279277 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.279546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.279448 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srnq\" (UniqueName: \"kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.279546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.279524 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.280107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.280061 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.282210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.282178 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.343615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.343527 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:29.344820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.344744 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srnq\" (UniqueName: \"kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq\") pod \"maas-api-revoke-sub-3921a4a6-kwj8g\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.399212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.399172 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:29.447218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.447186 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:29.447343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.447246 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:29.537367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.537133 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:29.844362 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:29.844332 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c961755_b435_426b_a6aa_c26a3c479d5c.slice/crio-561a5cdf246927e707df36e5a1dcaadbb8e832510460eab36bfccb8fd44341d4 WatchSource:0}: Error finding container 561a5cdf246927e707df36e5a1dcaadbb8e832510460eab36bfccb8fd44341d4: Status 404 returned error can't find the container with id 561a5cdf246927e707df36e5a1dcaadbb8e832510460eab36bfccb8fd44341d4 Oct 31 03:41:29.846953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:29.846927 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g"] Oct 31 03:41:30.069112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.027131 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:30.196125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.195952 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" event={"ID":"6c961755-b435-426b-a6aa-c26a3c479d5c","Type":"ContainerStarted","Data":"561a5cdf246927e707df36e5a1dcaadbb8e832510460eab36bfccb8fd44341d4"} Oct 31 03:41:30.285954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.285918 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:30.449619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.449511 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:30.453409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.451379 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:30.488654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.488621 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:30.561243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.558300 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf"] Oct 31 03:41:30.563406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.563375 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:30.577729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.577685 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a112b709-d618-4b06-9592-1b9983b56daa" containerName="revoke-keys" Oct 31 03:41:30.578002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.577977 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a112b709-d618-4b06-9592-1b9983b56daa" containerName="revoke-keys" Oct 31 03:41:30.578174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.578153 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca" pod "a112b709-d618-4b06-9592-1b9983b56daa" (UID: "a112b709-d618-4b06-9592-1b9983b56daa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:30.578253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.577767 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca\") pod \"a112b709-d618-4b06-9592-1b9983b56daa\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " Oct 31 03:41:30.578321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.578310 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj\" (UniqueName: \"kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj\") pod \"a112b709-d618-4b06-9592-1b9983b56daa\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " Oct 31 03:41:30.578434 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.578423 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca\") pod \"a112b709-d618-4b06-9592-1b9983b56daa\" (UID: \"a112b709-d618-4b06-9592-1b9983b56daa\") " Oct 31 03:41:30.578668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.578651 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a112b709-d618-4b06-9592-1b9983b56daa" containerName="revoke-keys" Oct 31 03:41:30.579001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.578984 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:30.582796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.582198 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca" pod "a112b709-d618-4b06-9592-1b9983b56daa" (UID: "a112b709-d618-4b06-9592-1b9983b56daa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:30.582954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.582857 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj" pod "a112b709-d618-4b06-9592-1b9983b56daa" (UID: "a112b709-d618-4b06-9592-1b9983b56daa"). InnerVolumeSpecName "kube-api-access-tzfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:30.584363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.584341 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf"] Oct 31 03:41:30.584546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.584428 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.679805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.679768 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.680002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.679830 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnpc\" (UniqueName: \"kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.680137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.680114 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.680315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.680302 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfdj\" (UniqueName: \"kubernetes.io/projected/a112b709-d618-4b06-9592-1b9983b56daa-kube-api-access-tzfdj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:30.680357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.680320 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a112b709-d618-4b06-9592-1b9983b56daa-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:30.781903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.781807 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.781903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.781875 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnpc\" (UniqueName: \"kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.782174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.781998 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.782687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.782667 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.784616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.784588 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.844430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.844402 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnpc\" (UniqueName: \"kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc\") pod \"maas-api-revoke-sub-33960ec2-k5ndf\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:30.949750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:30.949709 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:31.231197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.231160 4488 generic.go:356] "Generic (PLEG): container finished" podID="6c961755-b435-426b-a6aa-c26a3c479d5c" containerID="210b4b750414606ef602718d4f3461e135109e380d93886047ee3af5038a46d3" exitCode=0 Oct 31 03:41:31.231359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.231244 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" event={"ID":"6c961755-b435-426b-a6aa-c26a3c479d5c","Type":"ContainerDied","Data":"210b4b750414606ef602718d4f3461e135109e380d93886047ee3af5038a46d3"} Oct 31 03:41:31.233276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.233255 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" Oct 31 03:41:31.233431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.233275 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84" event={"ID":"a112b709-d618-4b06-9592-1b9983b56daa","Type":"ContainerDied","Data":"6a89db91c569b9e32524bc43ef2189681632ebf029e32179fd27de2617ed6ae9"} Oct 31 03:41:31.233431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.233318 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a89db91c569b9e32524bc43ef2189681632ebf029e32179fd27de2617ed6ae9" Oct 31 03:41:31.364449 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:31.364412 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebfbb64_06bb_4cb2_aa80_8cb14f6138db.slice/crio-5fac7021b238d999fd7395daea0293e8683cee11a7d986e34b89c22790532570 WatchSource:0}: Error finding container 5fac7021b238d999fd7395daea0293e8683cee11a7d986e34b89c22790532570: Status 404 returned error can't find the container with id 5fac7021b238d999fd7395daea0293e8683cee11a7d986e34b89c22790532570 Oct 31 03:41:31.413057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.404285 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf"] Oct 31 03:41:31.447987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.447952 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:31.448283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.448012 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:31.882873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:31.882811 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:32.243982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.243861 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" event={"ID":"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db","Type":"ContainerStarted","Data":"1e18b41febb6a01ddc3a707e590d2eeba153ac52f538a777839b8abee63dc0da"} Oct 31 03:41:32.243982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.243949 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" event={"ID":"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db","Type":"ContainerStarted","Data":"5fac7021b238d999fd7395daea0293e8683cee11a7d986e34b89c22790532570"} Oct 31 03:41:32.372139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.369464 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" podStartSLOduration=2.369444114 podStartE2EDuration="2.369444114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:32.359421998 +0000 UTC m=+1320.156140258" watchObservedRunningTime="2026-10-31 03:41:32.369444114 +0000 UTC m=+1320.166162372" Oct 31 03:41:32.454969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.451126 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:32.454969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.451193 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:32.589641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.589613 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:32.640413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.640380 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq\" (UniqueName: \"kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq\") pod \"6c961755-b435-426b-a6aa-c26a3c479d5c\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " Oct 31 03:41:32.640644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.640441 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca\") pod \"6c961755-b435-426b-a6aa-c26a3c479d5c\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " Oct 31 03:41:32.640723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.640676 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca\") pod \"6c961755-b435-426b-a6aa-c26a3c479d5c\" (UID: \"6c961755-b435-426b-a6aa-c26a3c479d5c\") " Oct 31 03:41:32.641072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.641039 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca" pod "6c961755-b435-426b-a6aa-c26a3c479d5c" (UID: "6c961755-b435-426b-a6aa-c26a3c479d5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:32.641498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.641473 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:32.642949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.642925 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca" pod "6c961755-b435-426b-a6aa-c26a3c479d5c" (UID: "6c961755-b435-426b-a6aa-c26a3c479d5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:32.643424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.643401 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq" pod "6c961755-b435-426b-a6aa-c26a3c479d5c" (UID: "6c961755-b435-426b-a6aa-c26a3c479d5c"). InnerVolumeSpecName "kube-api-access-6srnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:32.743580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.743454 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c961755-b435-426b-a6aa-c26a3c479d5c-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:32.743580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:32.743492 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6srnq\" (UniqueName: \"kubernetes.io/projected/6c961755-b435-426b-a6aa-c26a3c479d5c-kube-api-access-6srnq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:33.253752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.253720 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" Oct 31 03:41:33.254281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.253763 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g" event={"ID":"6c961755-b435-426b-a6aa-c26a3c479d5c","Type":"ContainerDied","Data":"561a5cdf246927e707df36e5a1dcaadbb8e832510460eab36bfccb8fd44341d4"} Oct 31 03:41:33.254281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.253856 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561a5cdf246927e707df36e5a1dcaadbb8e832510460eab36bfccb8fd44341d4" Oct 31 03:41:33.255756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.255725 4488 generic.go:356] "Generic (PLEG): container finished" podID="4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" containerID="1e18b41febb6a01ddc3a707e590d2eeba153ac52f538a777839b8abee63dc0da" exitCode=0 Oct 31 03:41:33.255891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.255815 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" event={"ID":"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db","Type":"ContainerDied","Data":"1e18b41febb6a01ddc3a707e590d2eeba153ac52f538a777839b8abee63dc0da"} Oct 31 03:41:33.447211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.447175 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:33.447362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.447233 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:33.538124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.537999 4488 scope.go:122] "RemoveContainer" containerID="54a9e842b51a08bb86b7bb4e6b17b5299982130ac7d02c6f61b55e072c112ffb" Oct 31 03:41:33.550820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.550793 4488 scope.go:122] "RemoveContainer" containerID="7d11f9034af08adec2c3dbd621099da37a07e7563b7480ea4c2f8764d0650223" Oct 31 03:41:33.563650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.563616 4488 scope.go:122] "RemoveContainer" containerID="de97ef28f27127068d11a1a57732cdea9e50437e20f4fe98ed7db6f4fdfcd7c1" Oct 31 03:41:33.573860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.573838 4488 scope.go:122] "RemoveContainer" containerID="8edcb717382271171ad8f1a34e09d8148d5991ef24f5cf541d9da49a06e00cfa" Oct 31 03:41:33.584330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.584310 4488 scope.go:122] "RemoveContainer" containerID="d67e6db46e6e506d81837aee4803cf4dee5e4c8bdda3ae56c3388f463e282e21" Oct 31 03:41:33.595317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.595296 4488 scope.go:122] "RemoveContainer" containerID="618db120e6514d284694f56a7f2fd65dcb4dfe2a5f4699776761d14d3ab6718b" Oct 31 03:41:33.631761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.631735 4488 scope.go:122] "RemoveContainer" containerID="06e8abebe26d66e839e48bfb85bbe20407477c0b16a49f97879ed1c1d9bc8069" Oct 31 03:41:33.645128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.644379 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:33.646613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.646153 4488 scope.go:122] "RemoveContainer" containerID="c3d302c5ac2ba70a1039f018e830c62964f602065c18b08c25f8ca6ed4c5b5e6" Oct 31 03:41:33.660246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.660222 4488 scope.go:122] "RemoveContainer" containerID="5d8bd624467e0d4707dbb650c4b8250a1da0369f99a5bdbb3168ac60e0cd8a5e" Oct 31 03:41:33.671409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.671390 4488 scope.go:122] "RemoveContainer" containerID="81fb2ad5b258932dee692864f996e45eafcb288c4ef8e0505db5e8306c62beea" Oct 31 03:41:33.830033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.830004 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:33.986352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:33.986322 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:34.434836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.434808 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:34.446916 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.446887 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:41:34.447081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.446937 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:41:34.470474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.470437 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca\") pod \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " Oct 31 03:41:34.470651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.470533 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc\" (UniqueName: \"kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc\") pod \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " Oct 31 03:41:34.470731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.470653 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca\") pod \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\" (UID: \"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db\") " Oct 31 03:41:34.471029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.471002 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca" pod "4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" (UID: "4ebfbb64-06bb-4cb2-aa80-8cb14f6138db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:34.471307 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.471284 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:34.472929 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.472859 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca" pod "4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" (UID: "4ebfbb64-06bb-4cb2-aa80-8cb14f6138db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:34.473070 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.473012 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc" pod "4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" (UID: "4ebfbb64-06bb-4cb2-aa80-8cb14f6138db"). InnerVolumeSpecName "kube-api-access-9tnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:34.573749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.573664 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:34.573749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:34.573706 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnpc\" (UniqueName: \"kubernetes.io/projected/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db-kube-api-access-9tnpc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:35.274066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.274033 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" Oct 31 03:41:35.274066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.274054 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf" event={"ID":"4ebfbb64-06bb-4cb2-aa80-8cb14f6138db","Type":"ContainerDied","Data":"5fac7021b238d999fd7395daea0293e8683cee11a7d986e34b89c22790532570"} Oct 31 03:41:35.274298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.274142 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fac7021b238d999fd7395daea0293e8683cee11a7d986e34b89c22790532570" Oct 31 03:41:35.451350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.451316 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:35.451865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.451843 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:35.452591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.452574 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:41:35.630002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.629961 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:35.826883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:35.826848 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:36.592637 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:41:36.592598 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:41:37.261448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:37.261410 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:38.048009 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.047971 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx"] Oct 31 03:41:38.050554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.048997 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6c961755-b435-426b-a6aa-c26a3c479d5c" containerName="revoke-keys" Oct 31 03:41:38.050554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.049027 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c961755-b435-426b-a6aa-c26a3c479d5c" containerName="revoke-keys" Oct 31 03:41:38.050554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.049083 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" containerName="revoke-keys" Oct 31 03:41:38.050554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.049124 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" containerName="revoke-keys" Oct 31 03:41:38.050554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.049437 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" containerName="revoke-keys" Oct 31 03:41:38.050554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.049470 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c961755-b435-426b-a6aa-c26a3c479d5c" containerName="revoke-keys" Oct 31 03:41:38.054200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.054176 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.093629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.093599 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.093814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.093657 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbv9\" (UniqueName: \"kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.093898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.093858 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.111338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.110592 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx"] Oct 31 03:41:38.153657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.153619 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:38.153657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.153651 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:41:38.153902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.153670 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:41:38.158228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.158210 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:41:38.195600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.195555 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.195825 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.195628 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbv9\" (UniqueName: \"kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.195825 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.195703 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.196455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.196428 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.198124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.198104 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.257559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.257531 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbv9\" (UniqueName: \"kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9\") pod \"maas-api-revoke-sub-64a0e4ab-rmnhx\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.373546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.373511 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:38.774460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:38.774432 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx"] Oct 31 03:41:38.774659 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:38.774638 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded534e73_0c88_4c25_8dc9_b0a5e292c234.slice/crio-e1bebcdfbcd64d2a362c9e8bb6bc03fa7c18470fae69fb75b0ca5566fbb3ce46 WatchSource:0}: Error finding container e1bebcdfbcd64d2a362c9e8bb6bc03fa7c18470fae69fb75b0ca5566fbb3ce46: Status 404 returned error can't find the container with id e1bebcdfbcd64d2a362c9e8bb6bc03fa7c18470fae69fb75b0ca5566fbb3ce46 Oct 31 03:41:39.373721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:39.373683 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" event={"ID":"ed534e73-0c88-4c25-8dc9-b0a5e292c234","Type":"ContainerStarted","Data":"5df63d5fd39083fe0062a6b4fc062b764dc846644c1897f00eb39e08258c1890"} Oct 31 03:41:39.374192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:39.373748 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" event={"ID":"ed534e73-0c88-4c25-8dc9-b0a5e292c234","Type":"ContainerStarted","Data":"e1bebcdfbcd64d2a362c9e8bb6bc03fa7c18470fae69fb75b0ca5566fbb3ce46"} Oct 31 03:41:39.491444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:39.491387 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" podStartSLOduration=1.49137186 podStartE2EDuration="1.49137186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:39.489230943 +0000 UTC m=+1327.285949203" watchObservedRunningTime="2026-10-31 03:41:39.49137186 +0000 UTC m=+1327.288090152" Oct 31 03:41:40.384565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:40.384533 4488 generic.go:356] "Generic (PLEG): container finished" podID="ed534e73-0c88-4c25-8dc9-b0a5e292c234" containerID="5df63d5fd39083fe0062a6b4fc062b764dc846644c1897f00eb39e08258c1890" exitCode=0 Oct 31 03:41:40.385016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:40.384623 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" event={"ID":"ed534e73-0c88-4c25-8dc9-b0a5e292c234","Type":"ContainerDied","Data":"5df63d5fd39083fe0062a6b4fc062b764dc846644c1897f00eb39e08258c1890"} Oct 31 03:41:40.538794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:40.538737 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:40.708879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:40.708781 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:40.964447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:40.964342 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf"] Oct 31 03:41:40.970065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:40.970028 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.033915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.033635 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf"] Oct 31 03:41:41.070685 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.070640 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.070867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.070705 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6g9p\" (UniqueName: \"kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p\") pod \"payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.076151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.076075 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9e38c94c-dockercfg-cwbq7\"" Oct 31 03:41:41.076316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.076075 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9e38c94c\"" Oct 31 03:41:41.172846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.172814 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.173082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.173060 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6g9p\" (UniqueName: \"kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p\") pod \"payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.174833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.174805 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.271163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.270857 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6g9p\" (UniqueName: \"kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p\") pod \"payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.302189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.300396 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:41.365837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.365763 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2"] Oct 31 03:41:41.372312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.372283 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.440278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.440243 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2"] Oct 31 03:41:41.480342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.480300 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.480744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.480720 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.480955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.480938 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.481223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.481205 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.481419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.481402 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7cx\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.481605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.481589 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.481790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.481774 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.482177 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.482081 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.482344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.482328 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584524 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584572 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584630 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584752 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584800 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584872 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584936 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7cx\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.584949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.584938 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.585057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.585013 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.585057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.585051 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.585185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.585060 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.585185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.585144 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.585452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.585431 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.585718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.585696 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.587897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.587869 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.588353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.588335 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.620397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.620368 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:41.646872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.646843 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.647021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.646908 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7cx\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-smtn2\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.686491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.686451 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca\") pod \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " Oct 31 03:41:41.686667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.686532 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca\") pod \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " Oct 31 03:41:41.686863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.686838 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca" pod "ed534e73-0c88-4c25-8dc9-b0a5e292c234" (UID: "ed534e73-0c88-4c25-8dc9-b0a5e292c234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:41.686942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.686865 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9\" (UniqueName: \"kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9\") pod \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\" (UID: \"ed534e73-0c88-4c25-8dc9-b0a5e292c234\") " Oct 31 03:41:41.687502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.687438 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:41.688820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.688799 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca" pod "ed534e73-0c88-4c25-8dc9-b0a5e292c234" (UID: "ed534e73-0c88-4c25-8dc9-b0a5e292c234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:41.688949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.688935 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9" pod "ed534e73-0c88-4c25-8dc9-b0a5e292c234" (UID: "ed534e73-0c88-4c25-8dc9-b0a5e292c234"). InnerVolumeSpecName "kube-api-access-jsbv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:41.698322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.698298 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:41.731235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.731190 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf"] Oct 31 03:41:41.731235 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:41.731206 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b6c560_d28d_4291_91a8_53137f45d971.slice/crio-af219c5d51ca8404b0602cb8a70dca392fd5446a64195366e717f1cce8a7652f WatchSource:0}: Error finding container af219c5d51ca8404b0602cb8a70dca392fd5446a64195366e717f1cce8a7652f: Status 404 returned error can't find the container with id af219c5d51ca8404b0602cb8a70dca392fd5446a64195366e717f1cce8a7652f Oct 31 03:41:41.789349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.789318 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbv9\" (UniqueName: \"kubernetes.io/projected/ed534e73-0c88-4c25-8dc9-b0a5e292c234-kube-api-access-jsbv9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:41.789577 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:41.789560 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed534e73-0c88-4c25-8dc9-b0a5e292c234-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:42.095151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.095122 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2"] Oct 31 03:41:42.095306 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:42.095281 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7949f7_535d_4204_aa14_28761776676f.slice/crio-922d0b8745b7d9d25a497596258c8a755ebf0ea89ebea573b2964ead424bc492 WatchSource:0}: Error finding container 922d0b8745b7d9d25a497596258c8a755ebf0ea89ebea573b2964ead424bc492: Status 404 returned error can't find the container with id 922d0b8745b7d9d25a497596258c8a755ebf0ea89ebea573b2964ead424bc492 Oct 31 03:41:42.097940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.097911 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:42.098016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.097979 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:42.098016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.098009 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:42.295538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.295507 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:42.412213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.412061 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" event={"ID":"67b6c560-d28d-4291-91a8-53137f45d971","Type":"ContainerStarted","Data":"be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246"} Oct 31 03:41:42.412494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.412453 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:42.412629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.412509 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" event={"ID":"67b6c560-d28d-4291-91a8-53137f45d971","Type":"ContainerStarted","Data":"af219c5d51ca8404b0602cb8a70dca392fd5446a64195366e717f1cce8a7652f"} Oct 31 03:41:42.414527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.414499 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" Oct 31 03:41:42.414659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.414586 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx" event={"ID":"ed534e73-0c88-4c25-8dc9-b0a5e292c234","Type":"ContainerDied","Data":"e1bebcdfbcd64d2a362c9e8bb6bc03fa7c18470fae69fb75b0ca5566fbb3ce46"} Oct 31 03:41:42.414729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.414663 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1bebcdfbcd64d2a362c9e8bb6bc03fa7c18470fae69fb75b0ca5566fbb3ce46" Oct 31 03:41:42.417511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.417481 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" event={"ID":"3c7949f7-535d-4204-aa14-28761776676f","Type":"ContainerStarted","Data":"e8768d2d3c7991260e8ed0a01817013ea86b748ec65ae3de96983a1619edac69"} Oct 31 03:41:42.417642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.417541 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" event={"ID":"3c7949f7-535d-4204-aa14-28761776676f","Type":"ContainerStarted","Data":"922d0b8745b7d9d25a497596258c8a755ebf0ea89ebea573b2964ead424bc492"} Oct 31 03:41:42.543218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.543169 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" podStartSLOduration=2.543155004 podStartE2EDuration="2.543155004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:42.538243186 +0000 UTC m=+1330.334961457" watchObservedRunningTime="2026-10-31 03:41:42.543155004 +0000 UTC m=+1330.339873261" Oct 31 03:41:42.611328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.611268 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:42.698441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.698357 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:42.715531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.715463 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podStartSLOduration=1.715439927 podStartE2EDuration="1.715439927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:42.712748049 +0000 UTC m=+1330.509466323" watchObservedRunningTime="2026-10-31 03:41:42.715439927 +0000 UTC m=+1330.512158185" Oct 31 03:41:42.829115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:42.822570 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:43.700294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:43.700259 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:43.700723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:43.700330 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:43.899739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:43.899703 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:44.065934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:44.065845 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:44.229130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:44.225144 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:44.699747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:44.699710 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:44.699957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:44.699765 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:44.730071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:44.730029 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:45.040770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:45.040680 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:45.194349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:45.194319 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:45.531551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:45.530891 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:45.750115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:45.724571 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:45.750115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:45.724635 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:45.750115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:45.739778 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:46.300128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.297591 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx"] Oct 31 03:41:46.300128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.298536 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ed534e73-0c88-4c25-8dc9-b0a5e292c234" containerName="revoke-keys" Oct 31 03:41:46.300128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.298556 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed534e73-0c88-4c25-8dc9-b0a5e292c234" containerName="revoke-keys" Oct 31 03:41:46.300128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.298745 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed534e73-0c88-4c25-8dc9-b0a5e292c234" containerName="revoke-keys" Oct 31 03:41:46.303590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.303562 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.370530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.370480 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx"] Oct 31 03:41:46.399698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.399661 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srvx8\" (UniqueName: \"kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.400941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.400917 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.401694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.401671 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.401925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.401488 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:41:46.402052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.401444 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:41:46.415134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.415104 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:41:46.415485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.415454 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:41:46.503597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.503553 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.503818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.503628 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.503818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.503758 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srvx8\" (UniqueName: \"kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.533217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.533178 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.536429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.536399 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.565589 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.565507 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srvx8\" (UniqueName: \"kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8\") pod \"maas-api-revoke-sub-e617087b-tshwx\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.680870 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.680834 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:46.699588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.699557 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:46.699714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:46.699622 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:47.187947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:47.187887 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx"] Oct 31 03:41:47.203349 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:41:47.202652 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb267f797_7931_4277_9b98_baa6d8f9105a.slice/crio-36eee33c218406b06bc7da7a74139a73a6563d8ef13892866c7d760d78f01b09 WatchSource:0}: Error finding container 36eee33c218406b06bc7da7a74139a73a6563d8ef13892866c7d760d78f01b09: Status 404 returned error can't find the container with id 36eee33c218406b06bc7da7a74139a73a6563d8ef13892866c7d760d78f01b09 Oct 31 03:41:47.248022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:47.247987 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:47.505415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:47.505314 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" event={"ID":"b267f797-7931-4277-9b98-baa6d8f9105a","Type":"ContainerStarted","Data":"36eee33c218406b06bc7da7a74139a73a6563d8ef13892866c7d760d78f01b09"} Oct 31 03:41:47.705606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:47.705562 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:47.705936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:47.705900 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:48.424353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.424303 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:48.462112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.461790 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:48.482152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.462776 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:48.547466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.519646 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:48.547466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.541188 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:48.606115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.604473 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:48.632587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.626302 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:48.632587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.626731 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:41:48.746175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.745378 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:48.746175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.745443 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:48.818114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.817851 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:48.910117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:48.909186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:41:49.159154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.153401 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.189118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.177260 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.189118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.185476 4488 generic.go:356] "Generic (PLEG): container finished" podID="b267f797-7931-4277-9b98-baa6d8f9105a" containerID="4906a461893d4af21597e63875a495dcb33b6568105dcc4f1c228f47b99c0b7d" exitCode=0 Oct 31 03:41:49.189118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.185855 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" event={"ID":"b267f797-7931-4277-9b98-baa6d8f9105a","Type":"ContainerDied","Data":"4906a461893d4af21597e63875a495dcb33b6568105dcc4f1c228f47b99c0b7d"} Oct 31 03:41:49.449220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.441577 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.469111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.453553 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:49.469111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.463321 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.469111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.465367 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.469450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.469247 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:41:49.484118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.479415 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.490427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.488582 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:41:49.490427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.488666 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:41:49.520242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.520195 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.528124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.522246 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:41:49.548115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.539719 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:49.571126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.569314 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.571126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.571061 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 31 03:41:49.571425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.571224 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:41:49.628117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.607189 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.628117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.616905 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:41:49.628117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.617865 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:49.659134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.628530 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.659134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.628896 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:41:49.659134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.649182 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:49.753183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.752387 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:49.753183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.752450 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:49.771116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.753836 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:49.771116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.753901 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.831122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.821772 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.831122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.823061 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:41:49.951787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.951748 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:49.953789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:49.953744 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:41:50.188507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.187740 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.259587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.188848 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:41:50.499190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.496304 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:50.566807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.521604 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.566807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.522795 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:50.566807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.523043 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:50.566807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.524592 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:50.566807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.524790 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:41:50.811202 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.578354 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:50.829433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.606433 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:41:50.830059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.830032 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:41:50.832689 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.606480 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:50.845117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.630853 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.850726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.630892 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:50.862128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.862078 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:41:50.862582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.862564 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:41:50.869310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.630984 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.886181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.651595 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.886772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.681744 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.888043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.700021 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:50.888269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.888241 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:50.888395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.833232 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:50.888862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.888839 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:41:50.971118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:50.958436 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:41:51.194739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.194689 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.369141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.349030 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:51.369141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.349121 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:51.537321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.537216 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:51.583076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.538891 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:51.600022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.538924 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:51.634986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.538949 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:51.634986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.539000 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 31 03:41:51.634986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.539020 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:51.634986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.539038 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:41:51.652262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.650805 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:51.652262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:51.650872 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.029135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.021685 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.303175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.034213 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:41:52.314119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.303809 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:41:52.558180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.117510 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.558180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.117571 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:52.558180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.117593 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:52.558180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.118750 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.579247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.118785 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:41:52.583117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.142238 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:52.637846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.162463 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:52.638623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.237291 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.638888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.045415 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:52.639071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.639035 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:52.748799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.718944 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:52.811381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.801929 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:52.811381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.802416 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:52.811381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.802750 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:52.811381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.803016 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:52.917118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.891425 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:41:53.247116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.245695 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:41:53.247116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.082789 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:53.247116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:52.915477 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": context deadline exceeded" Oct 31 03:41:53.352069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.282875 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:53.352069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.333256 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.479117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.453609 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:41:53.696901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.542687 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:53.697559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.581773 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:53.697708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.697677 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.697837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.651164 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.698061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.646665 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.698294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.651314 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:53.698478 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.651381 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:41:53.698637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.651407 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.698797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.669461 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.721451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.721399 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.812027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.811966 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:53.835393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.825562 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:53.835719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.825615 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:53.838315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:53.825663 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:54.029911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.029781 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:54.080124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.067631 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:54.080124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.067698 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:54.346262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.346202 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:54.486129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.479215 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 31 03:41:54.486376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.485247 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:41:54.537109 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:41:54.525555 4488 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.205s" Oct 31 03:41:54.537109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.526404 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:41:54.597184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.597157 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:54.615302 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.615263 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:54.615650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.615604 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:54.705201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.705147 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:54.706955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.706859 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:41:54.708383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.708355 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:54.709928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.708524 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:41:54.710554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.708568 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:41:54.711997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.708604 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:41:54.713053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.713029 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:41:54.749973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.749934 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:41:54.750295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.750263 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:54.759596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.758171 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca\") pod \"b267f797-7931-4277-9b98-baa6d8f9105a\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " Oct 31 03:41:54.760363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.760341 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca\") pod \"b267f797-7931-4277-9b98-baa6d8f9105a\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " Oct 31 03:41:54.760740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.760724 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8\" (UniqueName: \"kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8\") pod \"b267f797-7931-4277-9b98-baa6d8f9105a\" (UID: \"b267f797-7931-4277-9b98-baa6d8f9105a\") " Oct 31 03:41:54.767776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.767749 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca" pod "b267f797-7931-4277-9b98-baa6d8f9105a" (UID: "b267f797-7931-4277-9b98-baa6d8f9105a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:54.841139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.839224 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:41:54.841139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.840103 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:41:54.841394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.841321 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:41:54.843768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.841823 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:54.978514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:54.961769 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:55.075461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.075435 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:41:55.075701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.075687 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:41:55.084115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.078448 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8" pod "b267f797-7931-4277-9b98-baa6d8f9105a" (UID: "b267f797-7931-4277-9b98-baa6d8f9105a"). InnerVolumeSpecName "kube-api-access-srvx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:55.093650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.084430 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca" pod "b267f797-7931-4277-9b98-baa6d8f9105a" (UID: "b267f797-7931-4277-9b98-baa6d8f9105a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:55.216317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.216287 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srvx8\" (UniqueName: \"kubernetes.io/projected/b267f797-7931-4277-9b98-baa6d8f9105a-kube-api-access-srvx8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:55.216665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.216642 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b267f797-7931-4277-9b98-baa6d8f9105a-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:41:55.255041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.254683 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:41:55.290032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.290001 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:41:55.347735 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.346834 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:41:55.358719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.356598 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:41:55.358719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.357899 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:41:55.364179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.362154 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:41:55.379143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.377920 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:41:55.383341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.383320 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr"] Oct 31 03:41:55.384488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.384464 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b267f797-7931-4277-9b98-baa6d8f9105a" containerName="revoke-keys" Oct 31 03:41:55.384646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.384634 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b267f797-7931-4277-9b98-baa6d8f9105a" containerName="revoke-keys" Oct 31 03:41:55.384924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.384912 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b267f797-7931-4277-9b98-baa6d8f9105a" containerName="revoke-keys" Oct 31 03:41:55.399503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.399481 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr"] Oct 31 03:41:55.399768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.399755 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.403630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.403611 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:41:55.410657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.409772 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.446811 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnc4\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.446866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.446905 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.447138 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.447173 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.447260 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.447454 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.447506 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.449130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.447599 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.454386 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.454362 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e9fd0a85-openshift-default-dockercfg-mlsxh\"" Oct 31 03:41:55.466407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.466382 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:55.467900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.467702 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:55.468712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.468693 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:41:55.486064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.485341 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:41:55.498340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.498314 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:41:55.505433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.505352 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:41:55.510108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.508571 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:41:55.522107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.520163 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" event={"ID":"b267f797-7931-4277-9b98-baa6d8f9105a","Type":"ContainerDied","Data":"36eee33c218406b06bc7da7a74139a73a6563d8ef13892866c7d760d78f01b09"} Oct 31 03:41:55.522107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.520204 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx" Oct 31 03:41:55.522107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.520219 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36eee33c218406b06bc7da7a74139a73a6563d8ef13892866c7d760d78f01b09" Oct 31 03:41:55.552824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.552700 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.552824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.552780 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.552925 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.553130 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnc4\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.553179 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.553235 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.553488 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.553552 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.554273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.553649 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.556556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.555689 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.557819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.556907 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.557819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.557202 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.561196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.561175 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.561824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.561803 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.586076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.584916 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.587135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.586964 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.616828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.616787 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.617042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.617016 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnc4\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4\") pod \"e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:55.741799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:55.741755 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:56.228977 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.228932 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr"] Oct 31 03:41:56.238004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.237958 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:56.238146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.238053 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:56.238260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.238236 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:41:56.599712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.599672 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:56.625275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.623748 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" event={"ID":"789a8caa-b323-4999-b729-3274d34287c5","Type":"ContainerStarted","Data":"2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986"} Oct 31 03:41:56.625275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.623810 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" event={"ID":"789a8caa-b323-4999-b729-3274d34287c5","Type":"ContainerStarted","Data":"6b4ec2da287d8068bcb38f1f9dd51b8418489a80746cdd7ffb034d8c033e75f2"} Oct 31 03:41:56.747114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:56.746231 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:57.749834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:57.749803 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:58.039829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:58.039720 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podStartSLOduration=6.039704788 podStartE2EDuration="6.039704788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:58.035662364 +0000 UTC m=+1345.832380622" watchObservedRunningTime="2026-10-31 03:41:58.039704788 +0000 UTC m=+1345.836423045" Oct 31 03:41:58.322030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:58.321951 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:58.644077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:58.644046 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:58.645052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:58.645026 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:41:59.379400 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:59.379368 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:59.535248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:59.535215 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:59.866979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:41:59.866935 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:00.289205 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:00.289082 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:00.479527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:00.479486 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:00.648250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:00.648221 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:01.239031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:01.238993 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:01.468061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:01.468025 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:02.856686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:02.856615 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:03.695717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:03.695685 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:03.868130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:03.865909 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:04.025805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.025704 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:04.139217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.139177 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5"] Oct 31 03:42:04.145961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.145923 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.209141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.206970 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e9fd0a85\"" Oct 31 03:42:04.209141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.208211 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5"] Oct 31 03:42:04.247565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.246865 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e9fd0a85-dockercfg-x2pf7\"" Oct 31 03:42:04.283742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.271744 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.283742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.271944 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8rt\" (UniqueName: \"kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt\") pod \"payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.415117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.406363 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.415117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.406529 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8rt\" (UniqueName: \"kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt\") pod \"payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.415117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.408219 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.475402 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.475301 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8rt\" (UniqueName: \"kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt\") pod \"payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:04.764256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:04.764220 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:05.155478 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.155448 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5"] Oct 31 03:42:05.674376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.674328 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:05.717401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.717356 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" event={"ID":"75c2c1d3-6b3a-40be-8127-e8bdaf691104","Type":"ContainerStarted","Data":"9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76"} Oct 31 03:42:05.717722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.717433 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" event={"ID":"75c2c1d3-6b3a-40be-8127-e8bdaf691104","Type":"ContainerStarted","Data":"26c505bf53382c477f60a6514f8cd6924d47f091e86e1c4d76e898c738bf3d46"} Oct 31 03:42:05.718081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.718062 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:05.836285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.836225 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" podStartSLOduration=1.836209455 podStartE2EDuration="1.836209455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:05.834597901 +0000 UTC m=+1353.631316159" watchObservedRunningTime="2026-10-31 03:42:05.836209455 +0000 UTC m=+1353.632927713" Oct 31 03:42:05.926511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:05.926420 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:06.085792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:06.085761 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:06.591233 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:42:06.591199 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:42:07.035643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:07.035532 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:08.907118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:08.907074 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:09.091336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:09.091274 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:09.248262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:09.248179 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:09.409159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:09.409077 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:11.590895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.590853 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h"] Oct 31 03:42:11.596065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.596044 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694229 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694281 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694453 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvght\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694490 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694526 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694562 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694678 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.694887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.694864 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.695231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.695212 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.706018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.705989 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h"] Oct 31 03:42:11.796970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.796905 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.796989 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797024 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797145 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvght\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797193 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797225 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797248 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797365 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797467 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797513 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797605 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797748 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.797920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.797897 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.798080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.798053 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.799712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.799688 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.799803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.799788 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.860405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.860299 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.860857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.860830 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvght\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-svt6h\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:11.911049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:11.911012 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:12.354637 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:42:12.354606 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ace7f3c_d341_49a2_977f_580f8b3085df.slice/crio-c0879608b9d3ef40dfd644b76f8499c56ec2654d78427de8926b968fa8342bb2 WatchSource:0}: Error finding container c0879608b9d3ef40dfd644b76f8499c56ec2654d78427de8926b968fa8342bb2: Status 404 returned error can't find the container with id c0879608b9d3ef40dfd644b76f8499c56ec2654d78427de8926b968fa8342bb2 Oct 31 03:42:12.354779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.354711 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h"] Oct 31 03:42:12.357482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.357443 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:42:12.357603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.357526 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:42:12.357603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.357574 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:42:12.777327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.777220 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" event={"ID":"3ace7f3c-d341-49a2-977f-580f8b3085df","Type":"ContainerStarted","Data":"29c4cdedae50c24f137830c086d0f0de0618bce9bcbd2f26e4acecd0dcc1afe2"} Oct 31 03:42:12.777748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.777326 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" event={"ID":"3ace7f3c-d341-49a2-977f-580f8b3085df","Type":"ContainerStarted","Data":"c0879608b9d3ef40dfd644b76f8499c56ec2654d78427de8926b968fa8342bb2"} Oct 31 03:42:12.901989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.901938 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podStartSLOduration=1.901922891 podStartE2EDuration="1.901922891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:12.897810346 +0000 UTC m=+1360.694528605" watchObservedRunningTime="2026-10-31 03:42:12.901922891 +0000 UTC m=+1360.698641148" Oct 31 03:42:12.911677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:12.911644 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:13.330867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:13.330788 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:13.931007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:13.930974 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:42:13.931477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:13.931039 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:42:14.354195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:14.354160 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:14.532168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:14.528437 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:14.688228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:14.688134 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:14.929724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:14.929692 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:42:14.930531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:14.930022 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:42:15.911976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:15.911510 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:42:15.911976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:15.911571 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:42:16.928116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:16.914559 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:42:16.928116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:16.914645 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:42:17.741164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:17.741018 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:42:17.926113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:17.925547 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:42:17.926113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:17.925626 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:42:18.911955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:18.911897 4488 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:42:18.912408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:18.911980 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:42:19.846961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:19.846924 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:19.929064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:19.928913 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:19.930309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:19.930279 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:19.931226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:19.931199 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:42:23.907535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:23.907502 4488 generic.go:356] "Generic (PLEG): container finished" podID="66682f45-69b8-4d48-8c8f-6769133ee137" containerID="d790209e63e67374862b124807d37c03e2594b40582283006761e56a0aeaca4f" exitCode=0 Oct 31 03:42:23.907969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:23.907577 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fd8mr-np954" event={"ID":"66682f45-69b8-4d48-8c8f-6769133ee137","Type":"ContainerDied","Data":"d790209e63e67374862b124807d37c03e2594b40582283006761e56a0aeaca4f"} Oct 31 03:42:24.153758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:24.153691 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:24.407670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:24.407638 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:25.030415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.030274 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:25.136575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.136546 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:42:25.177139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177074 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path\") pod \"66682f45-69b8-4d48-8c8f-6769133ee137\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " Oct 31 03:42:25.177304 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177228 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle\") pod \"66682f45-69b8-4d48-8c8f-6769133ee137\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " Oct 31 03:42:25.177378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177326 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle\") pod \"66682f45-69b8-4d48-8c8f-6769133ee137\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " Oct 31 03:42:25.177459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177433 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path" pod "66682f45-69b8-4d48-8c8f-6769133ee137" (UID: "66682f45-69b8-4d48-8c8f-6769133ee137"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:42:25.177621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177603 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss\" (UniqueName: \"kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss\") pod \"66682f45-69b8-4d48-8c8f-6769133ee137\" (UID: \"66682f45-69b8-4d48-8c8f-6769133ee137\") " Oct 31 03:42:25.177709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177684 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle" pod "66682f45-69b8-4d48-8c8f-6769133ee137" (UID: "66682f45-69b8-4d48-8c8f-6769133ee137"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:42:25.177791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.177689 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle" pod "66682f45-69b8-4d48-8c8f-6769133ee137" (UID: "66682f45-69b8-4d48-8c8f-6769133ee137"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:42:25.178472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.178450 4488 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66682f45-69b8-4d48-8c8f-6769133ee137-archives-path\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:42:25.178472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.178473 4488 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-trusted-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:42:25.178618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.178482 4488 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66682f45-69b8-4d48-8c8f-6769133ee137-service-ca-bundle\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:42:25.180913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.180884 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss" pod "66682f45-69b8-4d48-8c8f-6769133ee137" (UID: "66682f45-69b8-4d48-8c8f-6769133ee137"). InnerVolumeSpecName "kube-api-access-kwtss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:42:25.183746 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.183721 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:25.281154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.281079 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtss\" (UniqueName: \"kubernetes.io/projected/66682f45-69b8-4d48-8c8f-6769133ee137-kube-api-access-kwtss\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:42:25.339061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.339030 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:25.944465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.944433 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fd8mr-np954" Oct 31 03:42:25.944465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.944451 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fd8mr-np954" event={"ID":"66682f45-69b8-4d48-8c8f-6769133ee137","Type":"ContainerDied","Data":"62b9b721dc1b2d513ce6da371e3b066ad2492252c9a90e00f587379f62a7585d"} Oct 31 03:42:25.944821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:25.944512 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b9b721dc1b2d513ce6da371e3b066ad2492252c9a90e00f587379f62a7585d" Oct 31 03:42:27.282675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:27.282640 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:27.433256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:27.433225 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:29.851278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:29.851248 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:33.792082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:33.792045 4488 scope.go:122] "RemoveContainer" containerID="364c2acc3b1b840b1466f9cbc694f0e656173e2e43bfe51be061b9f558e1ffb4" Oct 31 03:42:33.804069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:33.804046 4488 scope.go:122] "RemoveContainer" containerID="5df21f92a67510e9f87ccf83a8163c37e5ad847efc6b1e75dc0f8dc45e6b374a" Oct 31 03:42:33.815276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:33.815255 4488 scope.go:122] "RemoveContainer" containerID="693ca5b3172ecd93c4f99cf728e6ba7ccd811600ff4897c9f1f41bd77bb36167" Oct 31 03:42:33.826236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:33.826215 4488 scope.go:122] "RemoveContainer" containerID="3ba10ae607e9d8d6a20117bf86d8f5c6d8529625299e3a469a9aa6e4ba54a534" Oct 31 03:42:33.844837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:33.844806 4488 scope.go:122] "RemoveContainer" containerID="826a4005f84b16506a95334ae68e7d10633bcc39195bc1a14fc87007d88acf75" Oct 31 03:42:34.712388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:34.712355 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:36.103680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:36.103647 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:42:36.592333 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:42:36.592299 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:42:38.352234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.344788 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 31 03:42:38.501265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.501210 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.502575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.502542 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.517270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.517208 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.518355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.517691 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.528310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.527194 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.559271 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.559218 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.664926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.664821 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.706117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.705231 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.706117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.705709 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:42:38.707124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.707057 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:38.812118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.811680 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:42:38.857118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.855627 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:42:38.857118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:38.856779 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.154259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.154201 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 31 03:42:39.275316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.274203 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.548312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.548198 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:42:39.573160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.561616 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.556680 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.556744 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.556782 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.585406 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.586119 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.556723 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.586393 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.586673 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:42:39.637159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.586713 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:42:39.807216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.665532 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.807867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.708220 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.831119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.811850 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:42:39.851314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.831455 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:42:39.864125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.708262 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:42:39.913120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.877293 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:42:40.072427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.708316 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.894222 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.812186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.210920 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.210984 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.895733 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:39.895765 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.000551 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.252990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.251575 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:42:40.405199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.359910 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.405199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.384602 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:42:40.521721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.521674 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" podUID="af3162c5-fd5c-49bc-97f6-0897c174cc11" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.17:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.681218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.594890 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.695116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.595286 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.945357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.601330 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:42:41.032114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.649476 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.062545 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.700946 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:42:41.064562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.719988 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.064801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.808925 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:42:41.065618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.828768 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 31 03:42:41.066928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.880779 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.094718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:40.900759 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:41.314987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.290831 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:42:41.333678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.333622 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.367163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.363620 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:42:41.389275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.381523 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.756160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.570431 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:42:41.756792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.570467 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:42:41.756971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.574150 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:42:41.757131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.579636 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:42:41.757268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.637956 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:41.764123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.736950 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:42:41.790061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.790029 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:42:41.819838 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.819806 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:42:41.995122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:41.985385 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:42:42.091636 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.043340 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.137465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.137414 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.151117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.064896 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.156194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.066377 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.161132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.066431 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:42:42.186240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.043385 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.261763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.261709 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 03:42:42.403174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.377493 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:42:42.463112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.381717 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.492113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.391618 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.781190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.780296 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.795124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.789461 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.831675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.830987 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:42.844027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.843992 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:42:42.844286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.844058 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:42:42.844286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.844107 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:42:42.844286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.844143 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:42:42.844286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.844171 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:42:42.844286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.844201 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:42:42.844286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.844233 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:42:42.853447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.853420 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:42:42.853720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.853706 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:42:42.864049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.864019 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:42:42.896918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.894291 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:42:42.896918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.895846 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:42:42.896918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.896426 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:42:42.897277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.897195 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:42:42.897277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:42.897260 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:42:45.567540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:45.567460 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:50.421962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:50.421924 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:51.523157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:51.523126 4488 generic.go:356] "Generic (PLEG): container finished" podID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerID="1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297" exitCode=0 Oct 31 03:42:51.523591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:51.523181 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" event={"ID":"d3722cbb-788e-403c-8e29-2430eb5d804b","Type":"ContainerDied","Data":"1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297"} Oct 31 03:42:51.523635 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:51.523626 4488 scope.go:122] "RemoveContainer" containerID="1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297" Oct 31 03:42:51.525429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:51.525395 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:42:51.525524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:51.525467 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:42:51.525524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:51.525501 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:42:52.538817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:52.538774 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" event={"ID":"d3722cbb-788e-403c-8e29-2430eb5d804b","Type":"ContainerStarted","Data":"3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f"} Oct 31 03:42:53.089258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:53.089225 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:42:53.090303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:53.090277 4488 patch_prober.go:29] interesting pod/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:42:53.090435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:53.090335 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 03:42:54.089584 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:54.089546 4488 patch_prober.go:29] interesting pod/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:42:54.090040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:54.089607 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 03:42:55.089001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:55.088964 4488 patch_prober.go:29] interesting pod/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:42:55.089221 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:55.089018 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 03:42:55.130388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:55.130346 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:42:55.193877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:55.193842 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:56.089744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:56.089707 4488 patch_prober.go:29] interesting pod/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:42:56.089943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:56.089779 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 03:42:57.089058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:57.089023 4488 patch_prober.go:29] interesting pod/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:42:57.089557 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:57.089082 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 03:42:58.093795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:58.093756 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:42:58.094704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:58.094681 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:42:59.855312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:42:59.855278 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:00.021909 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.021879 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:00.152142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.152024 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf"] Oct 31 03:43:00.154019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.153956 4488 cpu_manager.go:414] "containerMap: removing container" podUID="66682f45-69b8-4d48-8c8f-6769133ee137" containerName="insights-gathering" Oct 31 03:43:00.154019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.153990 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66682f45-69b8-4d48-8c8f-6769133ee137" containerName="insights-gathering" Oct 31 03:43:00.154445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.154241 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="66682f45-69b8-4d48-8c8f-6769133ee137" containerName="insights-gathering" Oct 31 03:43:00.157835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.157808 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.189840 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.189810 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:00.196834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.196804 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g"] Oct 31 03:43:00.200330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.200312 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.214143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.214065 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf"] Oct 31 03:43:00.241325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.241290 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4td2m\"" Oct 31 03:43:00.262706 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.262598 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g"] Oct 31 03:43:00.309948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.309912 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a068c85-58af-48e3-9fe0-55756d21fb67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.309948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.309948 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310045 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310060 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310243 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310306 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310366 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310412 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310440 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kw2l\" (UniqueName: \"kubernetes.io/projected/1a068c85-58af-48e3-9fe0-55756d21fb67-kube-api-access-2kw2l\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310460 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zms\" (UniqueName: \"kubernetes.io/projected/ee63cc13-5d3f-4f87-a8f8-2c7987300722-kube-api-access-42zms\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310593 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310498 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310636 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310678 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310774 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310838 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.310914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310875 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.310914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310901 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.311007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.310993 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.412138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412023 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kw2l\" (UniqueName: \"kubernetes.io/projected/1a068c85-58af-48e3-9fe0-55756d21fb67-kube-api-access-2kw2l\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.412138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412063 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zms\" (UniqueName: \"kubernetes.io/projected/ee63cc13-5d3f-4f87-a8f8-2c7987300722-kube-api-access-42zms\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412264 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412332 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412384 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.412483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412447 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412511 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412573 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412544 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.412622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412573 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412693 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412727 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.412839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412811 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412893 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a068c85-58af-48e3-9fe0-55756d21fb67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.412968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412917 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412927 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.412968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.412927 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413057 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.413198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413079 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.413198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413055 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413153 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.413198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413183 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413258 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413286 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413318 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.413470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413368 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413496 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413644 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a068c85-58af-48e3-9fe0-55756d21fb67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.413764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.413693 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.415589 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.415558 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.415719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.415653 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.415719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.415669 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.415866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.415850 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.474298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.474261 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee63cc13-5d3f-4f87-a8f8-2c7987300722-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.474481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.474262 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a068c85-58af-48e3-9fe0-55756d21fb67-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.474481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.474320 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zms\" (UniqueName: \"kubernetes.io/projected/ee63cc13-5d3f-4f87-a8f8-2c7987300722-kube-api-access-42zms\") pod \"maas-default-gateway-openshift-default-fd4b99879-xxm8g\" (UID: \"ee63cc13-5d3f-4f87-a8f8-2c7987300722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.474481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.474384 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kw2l\" (UniqueName: \"kubernetes.io/projected/1a068c85-58af-48e3-9fe0-55756d21fb67-kube-api-access-2kw2l\") pod \"maas-default-gateway-openshift-default-fd4b99879-kqkqf\" (UID: \"1a068c85-58af-48e3-9fe0-55756d21fb67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.477233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.477214 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:00.517263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.517228 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:00.921930 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:43:00.921896 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a068c85_58af_48e3_9fe0_55756d21fb67.slice/crio-4eb83f4e15685f2a88102741cc69e22513804a09b98ae0d11462cd856350d78a WatchSource:0}: Error finding container 4eb83f4e15685f2a88102741cc69e22513804a09b98ae0d11462cd856350d78a: Status 404 returned error can't find the container with id 4eb83f4e15685f2a88102741cc69e22513804a09b98ae0d11462cd856350d78a Oct 31 03:43:00.923193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.923171 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf"] Oct 31 03:43:00.924770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.924735 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:00.924849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.924802 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:00.924849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.924832 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:00.976768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.976738 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g"] Oct 31 03:43:00.989691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.989646 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:00.989796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.989744 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:00.989857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:00.989793 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:01.684490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:01.682851 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" event={"ID":"ee63cc13-5d3f-4f87-a8f8-2c7987300722","Type":"ContainerStarted","Data":"01ded7354bed48118f64c32b84fb13c444d0ade40acb65455cb1971a837ff70a"} Oct 31 03:43:01.684490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:01.682950 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" event={"ID":"ee63cc13-5d3f-4f87-a8f8-2c7987300722","Type":"ContainerStarted","Data":"9d555cf46a30c08a6d1a8d1a8c4d9f3ad0c1d98108ac1aef22c37825bb3d3f52"} Oct 31 03:43:01.685776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:01.685748 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" event={"ID":"1a068c85-58af-48e3-9fe0-55756d21fb67","Type":"ContainerStarted","Data":"463301f29b2b4858789d18e3d2e8f457d5a459873c9f1ca87b10cbe2f80ee40f"} Oct 31 03:43:01.685922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:01.685793 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" event={"ID":"1a068c85-58af-48e3-9fe0-55756d21fb67","Type":"ContainerStarted","Data":"4eb83f4e15685f2a88102741cc69e22513804a09b98ae0d11462cd856350d78a"} Oct 31 03:43:01.863220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:01.863148 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podStartSLOduration=1.842048024 podStartE2EDuration="1.842048024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:01.826350546 +0000 UTC m=+1409.623068804" watchObservedRunningTime="2026-10-31 03:43:01.842048024 +0000 UTC m=+1409.638766283" Oct 31 03:43:01.958181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:01.954371 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podStartSLOduration=1.9543506590000002 podStartE2EDuration="1.954350659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:01.951865851 +0000 UTC m=+1409.748584111" watchObservedRunningTime="2026-10-31 03:43:01.954350659 +0000 UTC m=+1409.751068920" Oct 31 03:43:02.478323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:02.478286 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:02.480615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:02.480570 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:02.480816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:02.480653 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:02.523152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:02.520912 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:02.523152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:02.521050 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:02.523152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:02.521119 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:03.477827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:03.477793 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:03.478254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:03.477863 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:03.532112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:03.522215 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:03.532112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:03.522277 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:04.478464 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:04.478424 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:04.478961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:04.478484 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:04.543117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:04.540267 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:04.543117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:04.540363 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:04.949117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:04.944424 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:05.477967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:05.477930 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:05.478228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:05.478206 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:05.523634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:05.523600 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:05.524247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:05.524212 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:06.478293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:06.478257 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:06.478484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:06.478338 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:06.523131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:06.522508 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:06.523131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:06.522581 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:06.592064 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:43:06.592023 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:43:07.478166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:07.478127 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:07.478341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:07.478200 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:07.518434 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:07.518395 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:07.518590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:07.518465 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:08.477605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:08.477560 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:43:08.478120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:08.477646 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:43:08.518276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:08.518240 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:43:08.518457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:08.518311 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 03:43:09.751331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:09.751284 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:09.922626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:09.921841 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:10.136533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.136498 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:10.137759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.137740 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:10.142239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.142211 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:10.143733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.143711 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:10.148332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.148304 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" Oct 31 03:43:10.165380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.163649 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" Oct 31 03:43:10.351884 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:10.351841 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:14.706226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:14.706188 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:16.121029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:16.120983 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:43:19.699432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:19.699393 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:23.706179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.706141 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx"] Oct 31 03:43:23.709696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.709678 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.765526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.765494 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx"] Oct 31 03:43:23.809012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.808979 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:43:23.809375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.809346 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:43:23.809498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.809462 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:43:23.809569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.809514 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:43:23.838752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.838720 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.838943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.838915 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.839034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.839019 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92rqx\" (UniqueName: \"kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.940829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.940791 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.941027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.940903 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92rqx\" (UniqueName: \"kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.941142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.941080 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.941537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.941511 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:23.943564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:23.943542 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:24.003076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:24.003004 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92rqx\" (UniqueName: \"kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx\") pod \"maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:24.023996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:24.023963 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:24.457996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:24.457959 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx"] Oct 31 03:43:24.458767 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:43:24.458744 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8d9eaa_993e_4ebc_ad59_6af678d3f9ac.slice/crio-e1f6f147c69e4133447f3112cf3efb35e2cbdcd6c6f5caa82bf9da9e6e22a928 WatchSource:0}: Error finding container e1f6f147c69e4133447f3112cf3efb35e2cbdcd6c6f5caa82bf9da9e6e22a928: Status 404 returned error can't find the container with id e1f6f147c69e4133447f3112cf3efb35e2cbdcd6c6f5caa82bf9da9e6e22a928 Oct 31 03:43:24.660396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:24.660352 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:25.166674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:25.166638 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" event={"ID":"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac","Type":"ContainerStarted","Data":"82445b6b271ab14abbbd5a3d2c2b51a544de301d17e713a4c4014a4c3a4038ee"} Oct 31 03:43:25.167170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:25.166701 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" event={"ID":"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac","Type":"ContainerStarted","Data":"e1f6f147c69e4133447f3112cf3efb35e2cbdcd6c6f5caa82bf9da9e6e22a928"} Oct 31 03:43:25.193526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:25.193494 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:25.299832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:25.299766 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" podStartSLOduration=2.29974207 podStartE2EDuration="2.29974207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:25.287496744 +0000 UTC m=+1433.084215002" watchObservedRunningTime="2026-10-31 03:43:25.29974207 +0000 UTC m=+1433.096460329" Oct 31 03:43:26.176654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:26.176614 4488 generic.go:356] "Generic (PLEG): container finished" podID="ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" containerID="82445b6b271ab14abbbd5a3d2c2b51a544de301d17e713a4c4014a4c3a4038ee" exitCode=0 Oct 31 03:43:26.177213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:26.176685 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" event={"ID":"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac","Type":"ContainerDied","Data":"82445b6b271ab14abbbd5a3d2c2b51a544de301d17e713a4c4014a4c3a4038ee"} Oct 31 03:43:26.700481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:26.700446 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:26.886133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:26.886072 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:27.399151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.399079 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:27.538981 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.538889 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca\") pod \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " Oct 31 03:43:27.539200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.539047 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca\") pod \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " Oct 31 03:43:27.539200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.539179 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx\" (UniqueName: \"kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx\") pod \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\" (UID: \"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac\") " Oct 31 03:43:27.539947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.539889 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca" pod "ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" (UID: "ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:27.544760 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.544732 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca" pod "ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" (UID: "ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:43:27.545178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.545155 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx" pod "ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" (UID: "ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac"). InnerVolumeSpecName "kube-api-access-92rqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:27.641283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.641243 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92rqx\" (UniqueName: \"kubernetes.io/projected/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-kube-api-access-92rqx\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:27.641283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.641281 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:27.641507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:27.641294 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:28.195627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:28.195599 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" Oct 31 03:43:28.195844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:28.195620 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx" event={"ID":"ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac","Type":"ContainerDied","Data":"e1f6f147c69e4133447f3112cf3efb35e2cbdcd6c6f5caa82bf9da9e6e22a928"} Oct 31 03:43:28.195955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:28.195870 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1f6f147c69e4133447f3112cf3efb35e2cbdcd6c6f5caa82bf9da9e6e22a928" Oct 31 03:43:29.835046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.835004 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l"] Oct 31 03:43:29.836783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.836752 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" containerName="revoke-keys" Oct 31 03:43:29.836783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.836784 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" containerName="revoke-keys" Oct 31 03:43:29.837246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.837227 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" containerName="revoke-keys" Oct 31 03:43:29.841867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.841842 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.881010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.880971 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.881254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.881224 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqrg\" (UniqueName: \"kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.881332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.881277 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.891666 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.891619 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l"] Oct 31 03:43:29.935137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.933394 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:43:29.935137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.933496 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:43:29.935137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.933605 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:43:29.947607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.947576 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:43:29.983297 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.983259 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqrg\" (UniqueName: \"kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.983506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.983320 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.983614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.983598 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.984333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.984314 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:29.985827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:29.985808 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:30.045036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:30.045000 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqrg\" (UniqueName: \"kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg\") pod \"maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:30.163890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:30.163821 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:30.561668 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:43:30.561635 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35127fff_11e4_4ee6_91e1_f818f1577570.slice/crio-d17572e1f5c7feb63cfeb0be18c11080c5fc8829b5d0a2543dc0247ae45810af WatchSource:0}: Error finding container d17572e1f5c7feb63cfeb0be18c11080c5fc8829b5d0a2543dc0247ae45810af: Status 404 returned error can't find the container with id d17572e1f5c7feb63cfeb0be18c11080c5fc8829b5d0a2543dc0247ae45810af Oct 31 03:43:30.564246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:30.564214 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l"] Oct 31 03:43:30.651419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:30.651384 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:31.222895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:31.222803 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" event={"ID":"35127fff-11e4-4ee6-91e1-f818f1577570","Type":"ContainerStarted","Data":"1fba6c2b04016b6de0c9a0d1c88a9d1aedfa1fa480336b79c22ad28ee549dbef"} Oct 31 03:43:31.222895 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:31.222866 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" event={"ID":"35127fff-11e4-4ee6-91e1-f818f1577570","Type":"ContainerStarted","Data":"d17572e1f5c7feb63cfeb0be18c11080c5fc8829b5d0a2543dc0247ae45810af"} Oct 31 03:43:31.340608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:31.340548 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" podStartSLOduration=2.34053128 podStartE2EDuration="2.34053128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:31.338957917 +0000 UTC m=+1439.135676174" watchObservedRunningTime="2026-10-31 03:43:31.34053128 +0000 UTC m=+1439.137249540" Oct 31 03:43:32.234192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:32.234156 4488 generic.go:356] "Generic (PLEG): container finished" podID="35127fff-11e4-4ee6-91e1-f818f1577570" containerID="1fba6c2b04016b6de0c9a0d1c88a9d1aedfa1fa480336b79c22ad28ee549dbef" exitCode=0 Oct 31 03:43:32.234729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:32.234242 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" event={"ID":"35127fff-11e4-4ee6-91e1-f818f1577570","Type":"ContainerDied","Data":"1fba6c2b04016b6de0c9a0d1c88a9d1aedfa1fa480336b79c22ad28ee549dbef"} Oct 31 03:43:33.493610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.493524 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:33.556965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.556921 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca\") pod \"35127fff-11e4-4ee6-91e1-f818f1577570\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " Oct 31 03:43:33.557242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.557005 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg\" (UniqueName: \"kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg\") pod \"35127fff-11e4-4ee6-91e1-f818f1577570\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " Oct 31 03:43:33.557242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.557049 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca\") pod \"35127fff-11e4-4ee6-91e1-f818f1577570\" (UID: \"35127fff-11e4-4ee6-91e1-f818f1577570\") " Oct 31 03:43:33.557506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.557474 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca" pod "35127fff-11e4-4ee6-91e1-f818f1577570" (UID: "35127fff-11e4-4ee6-91e1-f818f1577570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:33.557876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.557852 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:33.559687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.559661 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg" pod "35127fff-11e4-4ee6-91e1-f818f1577570" (UID: "35127fff-11e4-4ee6-91e1-f818f1577570"). InnerVolumeSpecName "kube-api-access-mfqrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:33.559801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.559719 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca" pod "35127fff-11e4-4ee6-91e1-f818f1577570" (UID: "35127fff-11e4-4ee6-91e1-f818f1577570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:43:33.660302 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.660252 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35127fff-11e4-4ee6-91e1-f818f1577570-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:33.660302 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:33.660296 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqrg\" (UniqueName: \"kubernetes.io/projected/35127fff-11e4-4ee6-91e1-f818f1577570-kube-api-access-mfqrg\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:34.253424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:34.253391 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" Oct 31 03:43:34.253613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:34.253429 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l" event={"ID":"35127fff-11e4-4ee6-91e1-f818f1577570","Type":"ContainerDied","Data":"d17572e1f5c7feb63cfeb0be18c11080c5fc8829b5d0a2543dc0247ae45810af"} Oct 31 03:43:34.253613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:34.253515 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17572e1f5c7feb63cfeb0be18c11080c5fc8829b5d0a2543dc0247ae45810af" Oct 31 03:43:35.284181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.277469 4488 generic.go:356] "Generic (PLEG): container finished" podID="8eb7b302-9057-446b-abf7-90c8071fb932" containerID="dd2a444b5219548823a821497f5d2ebeb1583b3001b0e8080e16691f8115f33f" exitCode=0 Oct 31 03:43:35.284181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.277509 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" event={"ID":"8eb7b302-9057-446b-abf7-90c8071fb932","Type":"ContainerDied","Data":"dd2a444b5219548823a821497f5d2ebeb1583b3001b0e8080e16691f8115f33f"} Oct 31 03:43:35.284181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.278058 4488 scope.go:122] "RemoveContainer" containerID="dd2a444b5219548823a821497f5d2ebeb1583b3001b0e8080e16691f8115f33f" Oct 31 03:43:35.299437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.299385 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:35.299599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.299475 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:35.299599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.299510 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:43:35.999710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:35.999679 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:36.288783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:36.288698 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" event={"ID":"8eb7b302-9057-446b-abf7-90c8071fb932","Type":"ContainerStarted","Data":"c2caa3a2b498efc00db1e3742710403a182faa78827ffe0b62eac563a92995b9"} Oct 31 03:43:36.590984 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:43:36.590951 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:43:36.987146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:36.987017 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:43:36.988546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:36.988511 4488 patch_prober.go:29] interesting pod/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:43:36.988694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:36.988568 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:43:37.849258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.849225 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h"] Oct 31 03:43:37.850117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.850067 4488 cpu_manager.go:414] "containerMap: removing container" podUID="35127fff-11e4-4ee6-91e1-f818f1577570" containerName="revoke-keys" Oct 31 03:43:37.850117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.850109 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35127fff-11e4-4ee6-91e1-f818f1577570" containerName="revoke-keys" Oct 31 03:43:37.850308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.850295 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="35127fff-11e4-4ee6-91e1-f818f1577570" containerName="revoke-keys" Oct 31 03:43:37.852986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.852951 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.868832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.868796 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q7lg\" (UniqueName: \"kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.868998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.868980 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.869169 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.869156 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.944076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.944004 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h"] Oct 31 03:43:37.944411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.944294 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:37.954783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.954752 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:43:37.954783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.954756 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:43:37.955026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.954797 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:43:37.970413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.970382 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q7lg\" (UniqueName: \"kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.970609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.970506 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.970682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.970617 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.971367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.971343 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.973170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.973077 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:37.987790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.987000 4488 patch_prober.go:29] interesting pod/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:43:37.987790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.987056 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:43:37.987790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:37.987681 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:43:38.076244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:38.076208 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q7lg\" (UniqueName: \"kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg\") pod \"maas-api-revoke-sub-9db2ec9e-plt2h\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:38.145749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:38.145308 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:38.169207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:38.169177 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:38.601917 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:43:38.601886 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e7f54a_3a9d_4720_b6d8_dfed1b2674be.slice/crio-814510dd1f0ac9605ae66b9eafae378e14fd03c706f6c1952a40fc8f69b31314 WatchSource:0}: Error finding container 814510dd1f0ac9605ae66b9eafae378e14fd03c706f6c1952a40fc8f69b31314: Status 404 returned error can't find the container with id 814510dd1f0ac9605ae66b9eafae378e14fd03c706f6c1952a40fc8f69b31314 Oct 31 03:43:38.602926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:38.602905 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h"] Oct 31 03:43:38.987873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:38.987798 4488 patch_prober.go:29] interesting pod/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:43:38.987873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:38.987856 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:43:39.470114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:39.469832 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" event={"ID":"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be","Type":"ContainerStarted","Data":"f524c0611f07f2ac184e42470fe8d12894a8f29347a205d405216bbf1df8ef1d"} Oct 31 03:43:39.470114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:39.469902 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" event={"ID":"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be","Type":"ContainerStarted","Data":"814510dd1f0ac9605ae66b9eafae378e14fd03c706f6c1952a40fc8f69b31314"} Oct 31 03:43:39.595332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:39.595273 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" podStartSLOduration=2.5952549510000003 podStartE2EDuration="2.595254951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:39.589707498 +0000 UTC m=+1447.386425754" watchObservedRunningTime="2026-10-31 03:43:39.595254951 +0000 UTC m=+1447.391973213" Oct 31 03:43:39.986941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:39.986917 4488 patch_prober.go:29] interesting pod/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:43:39.987058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:39.986966 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:43:40.483003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:40.482967 4488 generic.go:356] "Generic (PLEG): container finished" podID="d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" containerID="f524c0611f07f2ac184e42470fe8d12894a8f29347a205d405216bbf1df8ef1d" exitCode=0 Oct 31 03:43:40.483440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:40.483104 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" event={"ID":"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be","Type":"ContainerDied","Data":"f524c0611f07f2ac184e42470fe8d12894a8f29347a205d405216bbf1df8ef1d"} Oct 31 03:43:40.987421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:40.987386 4488 patch_prober.go:29] interesting pod/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:43:40.987635 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:40.987445 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 03:43:41.140857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.140826 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:43:41.320256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.319997 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:41.672366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.672338 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:41.751588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.751546 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca\") pod \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " Oct 31 03:43:41.751801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.751752 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg\" (UniqueName: \"kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg\") pod \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " Oct 31 03:43:41.751920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.751904 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca\") pod \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\" (UID: \"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be\") " Oct 31 03:43:41.752035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.751895 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca" pod "d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" (UID: "d0e7f54a-3a9d-4720-b6d8-dfed1b2674be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:41.752809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.752785 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.754084 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.754064 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca" pod "d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" (UID: "d0e7f54a-3a9d-4720-b6d8-dfed1b2674be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:43:41.754222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.754100 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg" pod "d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" (UID: "d0e7f54a-3a9d-4720-b6d8-dfed1b2674be"). InnerVolumeSpecName "kube-api-access-2q7lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:41.854880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.854845 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.855113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.854888 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q7lg\" (UniqueName: \"kubernetes.io/projected/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be-kube-api-access-2q7lg\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.991904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.991817 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:43:41.992939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:41.992915 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:43:42.510498 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.510462 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" Oct 31 03:43:42.510678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.510497 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h" event={"ID":"d0e7f54a-3a9d-4720-b6d8-dfed1b2674be","Type":"ContainerDied","Data":"814510dd1f0ac9605ae66b9eafae378e14fd03c706f6c1952a40fc8f69b31314"} Oct 31 03:43:42.510678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.510574 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="814510dd1f0ac9605ae66b9eafae378e14fd03c706f6c1952a40fc8f69b31314" Oct 31 03:43:42.723222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.723125 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf"] Oct 31 03:43:42.733591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.733562 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r"] Oct 31 03:43:42.734669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.734644 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" containerName="revoke-keys" Oct 31 03:43:42.734722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.734674 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" containerName="revoke-keys" Oct 31 03:43:42.734878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.734866 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" containerName="revoke-keys" Oct 31 03:43:42.738410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.738389 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.790963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.790934 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:43:42.791339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.791207 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:43:42.791480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.791292 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:43:42.796310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.796280 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r"] Oct 31 03:43:42.825972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.825940 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:43:42.873149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.873080 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.873379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.873359 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.873481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.873466 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.972682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.972649 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:42.975381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.975321 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.975485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.975436 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.975598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.975584 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.976245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.976223 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:42.978286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:42.978260 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:43.039010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:43.038926 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck\") pod \"maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:43.056855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:43.052444 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:43.188350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:43.155956 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:43.342750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:43.341491 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:43.588076 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:43:43.588040 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a02fe56_0ef1_4425_bd77_c52f6caf2976.slice/crio-002f55ec4fa0d22f1d5e64faf8daec616e1ccf4e2a848ce45ac9d674b2106fa8 WatchSource:0}: Error finding container 002f55ec4fa0d22f1d5e64faf8daec616e1ccf4e2a848ce45ac9d674b2106fa8: Status 404 returned error can't find the container with id 002f55ec4fa0d22f1d5e64faf8daec616e1ccf4e2a848ce45ac9d674b2106fa8 Oct 31 03:43:43.589026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:43.589000 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r"] Oct 31 03:43:44.552043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:44.552002 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" event={"ID":"1a02fe56-0ef1-4425-bd77-c52f6caf2976","Type":"ContainerStarted","Data":"6f43a2e6e6d10b0d7c0068704c6bd42e5aac3d80d1831008ddcf6b1041968ecd"} Oct 31 03:43:44.552592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:44.552073 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" event={"ID":"1a02fe56-0ef1-4425-bd77-c52f6caf2976","Type":"ContainerStarted","Data":"002f55ec4fa0d22f1d5e64faf8daec616e1ccf4e2a848ce45ac9d674b2106fa8"} Oct 31 03:43:44.672415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:44.672361 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" podStartSLOduration=2.672347134 podStartE2EDuration="2.672347134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:44.667871229 +0000 UTC m=+1452.464589490" watchObservedRunningTime="2026-10-31 03:43:44.672347134 +0000 UTC m=+1452.469065392" Oct 31 03:43:45.562697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:45.562661 4488 generic.go:356] "Generic (PLEG): container finished" podID="1a02fe56-0ef1-4425-bd77-c52f6caf2976" containerID="6f43a2e6e6d10b0d7c0068704c6bd42e5aac3d80d1831008ddcf6b1041968ecd" exitCode=0 Oct 31 03:43:45.563168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:45.562754 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" event={"ID":"1a02fe56-0ef1-4425-bd77-c52f6caf2976","Type":"ContainerDied","Data":"6f43a2e6e6d10b0d7c0068704c6bd42e5aac3d80d1831008ddcf6b1041968ecd"} Oct 31 03:43:46.403724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.403690 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:46.744702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.744676 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:46.805060 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.805016 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca\") pod \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " Oct 31 03:43:46.805449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.805416 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca\") pod \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " Oct 31 03:43:46.805594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.805466 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck\") pod \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\" (UID: \"1a02fe56-0ef1-4425-bd77-c52f6caf2976\") " Oct 31 03:43:46.805594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.805467 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca" pod "1a02fe56-0ef1-4425-bd77-c52f6caf2976" (UID: "1a02fe56-0ef1-4425-bd77-c52f6caf2976"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:46.806262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.806238 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:46.808193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.808147 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck" pod "1a02fe56-0ef1-4425-bd77-c52f6caf2976" (UID: "1a02fe56-0ef1-4425-bd77-c52f6caf2976"). InnerVolumeSpecName "kube-api-access-9mgck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:46.808322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.808263 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca" pod "1a02fe56-0ef1-4425-bd77-c52f6caf2976" (UID: "1a02fe56-0ef1-4425-bd77-c52f6caf2976"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:43:46.907875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.907839 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a02fe56-0ef1-4425-bd77-c52f6caf2976-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:46.907875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:46.907878 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/1a02fe56-0ef1-4425-bd77-c52f6caf2976-kube-api-access-9mgck\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:47.587165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:47.587135 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" Oct 31 03:43:47.587330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:47.587142 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r" event={"ID":"1a02fe56-0ef1-4425-bd77-c52f6caf2976","Type":"ContainerDied","Data":"002f55ec4fa0d22f1d5e64faf8daec616e1ccf4e2a848ce45ac9d674b2106fa8"} Oct 31 03:43:47.587330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:47.587279 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="002f55ec4fa0d22f1d5e64faf8daec616e1ccf4e2a848ce45ac9d674b2106fa8" Oct 31 03:43:47.724342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:47.724287 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" podUID="67b6c560-d28d-4291-91a8-53137f45d971" containerName="payload-pre-processing" containerID="cri-o://be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246" gracePeriod=25 Oct 31 03:43:48.050920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.050883 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:43:48.128633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.128535 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume\") pod \"67b6c560-d28d-4291-91a8-53137f45d971\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " Oct 31 03:43:48.128807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.128640 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p\" (UniqueName: \"kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p\") pod \"67b6c560-d28d-4291-91a8-53137f45d971\" (UID: \"67b6c560-d28d-4291-91a8-53137f45d971\") " Oct 31 03:43:48.129016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.128991 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume" pod "67b6c560-d28d-4291-91a8-53137f45d971" (UID: "67b6c560-d28d-4291-91a8-53137f45d971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:48.129472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.129456 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b6c560-d28d-4291-91a8-53137f45d971-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:48.130846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.130826 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p" pod "67b6c560-d28d-4291-91a8-53137f45d971" (UID: "67b6c560-d28d-4291-91a8-53137f45d971"). InnerVolumeSpecName "kube-api-access-m6g9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:48.231241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.231200 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6g9p\" (UniqueName: \"kubernetes.io/projected/67b6c560-d28d-4291-91a8-53137f45d971-kube-api-access-m6g9p\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:43:48.583722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.583687 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:48.597076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.597044 4488 generic.go:356] "Generic (PLEG): container finished" podID="67b6c560-d28d-4291-91a8-53137f45d971" containerID="be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246" exitCode=0 Oct 31 03:43:48.597274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.597133 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" event={"ID":"67b6c560-d28d-4291-91a8-53137f45d971","Type":"ContainerDied","Data":"be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246"} Oct 31 03:43:48.597274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.597159 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" Oct 31 03:43:48.597274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.597202 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf" event={"ID":"67b6c560-d28d-4291-91a8-53137f45d971","Type":"ContainerDied","Data":"af219c5d51ca8404b0602cb8a70dca392fd5446a64195366e717f1cce8a7652f"} Oct 31 03:43:48.597274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.597233 4488 scope.go:122] "RemoveContainer" containerID="be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246" Oct 31 03:43:48.608677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.608654 4488 scope.go:122] "RemoveContainer" containerID="be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246" Oct 31 03:43:48.608962 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:43:48.608944 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246\": container with ID starting with be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246 not found: ID does not exist" containerID="be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246" Oct 31 03:43:48.609007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.608972 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246"} err="failed to get container status \"be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246\": rpc error: code = NotFound desc = could not find container \"be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246\": container with ID starting with be21e192ce71a039b0a1c1c248c3f9a1c312220c21a81e8edea89e30b1831246 not found: ID does not exist" Oct 31 03:43:48.765969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.765931 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf"] Oct 31 03:43:48.815258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:48.815225 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e38c94c-b847b589f-v56mf"] Oct 31 03:43:49.862992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:49.862962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:50.318295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:50.318216 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b6c560-d28d-4291-91a8-53137f45d971" path="/var/lib/kubelet/pods/67b6c560-d28d-4291-91a8-53137f45d971/volumes" Oct 31 03:43:51.210390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:51.210361 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:54.684979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:54.684943 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:54.918628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:54.918499 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:55.093421 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:55.092886 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:56.118669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:56.118633 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:59.053519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:59.053475 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f"] Oct 31 03:43:59.106697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:43:59.106660 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd915c1f-g7c6f"] Oct 31 03:44:00.318554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:00.318515 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18799e14-97c1-4493-beea-610b5cac08c0" path="/var/lib/kubelet/pods/18799e14-97c1-4493-beea-610b5cac08c0/volumes" Oct 31 03:44:00.906877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:00.906841 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:06.590860 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:44:06.590823 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:44:07.197987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:07.197955 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:10.697169 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:10.697079 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:14.452225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.452118 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7"] Oct 31 03:44:14.453018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.452994 4488 cpu_manager.go:414] "containerMap: removing container" podUID="67b6c560-d28d-4291-91a8-53137f45d971" containerName="payload-pre-processing" Oct 31 03:44:14.453188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.453021 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b6c560-d28d-4291-91a8-53137f45d971" containerName="payload-pre-processing" Oct 31 03:44:14.453188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.453063 4488 cpu_manager.go:414] "containerMap: removing container" podUID="1a02fe56-0ef1-4425-bd77-c52f6caf2976" containerName="revoke-keys" Oct 31 03:44:14.453188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.453072 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a02fe56-0ef1-4425-bd77-c52f6caf2976" containerName="revoke-keys" Oct 31 03:44:14.453369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.453318 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b6c560-d28d-4291-91a8-53137f45d971" containerName="payload-pre-processing" Oct 31 03:44:14.453369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.453342 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a02fe56-0ef1-4425-bd77-c52f6caf2976" containerName="revoke-keys" Oct 31 03:44:14.456539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.456516 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.533118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.532531 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:44:14.536543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.536512 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7"] Oct 31 03:44:14.563899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.563866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.564244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.564211 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.564405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.564345 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njtj\" (UniqueName: \"kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.565695 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.565672 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:44:14.565816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.565716 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:44:14.565816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.565724 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:44:14.665951 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.665912 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.666167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.666151 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.666220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.666209 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njtj\" (UniqueName: \"kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.666824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.666761 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.668563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.668542 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.729819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.729741 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njtj\" (UniqueName: \"kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj\") pod \"maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:14.770503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:14.770471 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:15.183821 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:44:15.183781 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6526bc3c_b8e6_4604_b813_6860b62ddb0f.slice/crio-d43edbf0759ee18239209ebe1054059774705c000a92a4db35e9c7ec343ab685 WatchSource:0}: Error finding container d43edbf0759ee18239209ebe1054059774705c000a92a4db35e9c7ec343ab685: Status 404 returned error can't find the container with id d43edbf0759ee18239209ebe1054059774705c000a92a4db35e9c7ec343ab685 Oct 31 03:44:15.184985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:15.184960 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7"] Oct 31 03:44:15.844394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:15.844364 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:15.907852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:15.905829 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" event={"ID":"6526bc3c-b8e6-4604-b813-6860b62ddb0f","Type":"ContainerStarted","Data":"df92f25cc709a5940e20eaea7c070db0ec1f1abc4a61f704c688612a6f144837"} Oct 31 03:44:15.907852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:15.905899 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" event={"ID":"6526bc3c-b8e6-4604-b813-6860b62ddb0f","Type":"ContainerStarted","Data":"d43edbf0759ee18239209ebe1054059774705c000a92a4db35e9c7ec343ab685"} Oct 31 03:44:16.024145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:16.024050 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" podStartSLOduration=2.024030083 podStartE2EDuration="2.024030083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:16.022591058 +0000 UTC m=+1483.819309315" watchObservedRunningTime="2026-10-31 03:44:16.024030083 +0000 UTC m=+1483.820748343" Oct 31 03:44:16.919371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:16.919330 4488 generic.go:356] "Generic (PLEG): container finished" podID="6526bc3c-b8e6-4604-b813-6860b62ddb0f" containerID="df92f25cc709a5940e20eaea7c070db0ec1f1abc4a61f704c688612a6f144837" exitCode=0 Oct 31 03:44:16.919915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:16.919410 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" event={"ID":"6526bc3c-b8e6-4604-b813-6860b62ddb0f","Type":"ContainerDied","Data":"df92f25cc709a5940e20eaea7c070db0ec1f1abc4a61f704c688612a6f144837"} Oct 31 03:44:18.182677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.182603 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:18.336598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.336552 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca\") pod \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " Oct 31 03:44:18.336811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.336612 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj\" (UniqueName: \"kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj\") pod \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " Oct 31 03:44:18.336891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.336839 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca\") pod \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\" (UID: \"6526bc3c-b8e6-4604-b813-6860b62ddb0f\") " Oct 31 03:44:18.337157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.337053 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca" pod "6526bc3c-b8e6-4604-b813-6860b62ddb0f" (UID: "6526bc3c-b8e6-4604-b813-6860b62ddb0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:18.337866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.337836 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:18.339729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.339699 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca" pod "6526bc3c-b8e6-4604-b813-6860b62ddb0f" (UID: "6526bc3c-b8e6-4604-b813-6860b62ddb0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:44:18.339729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.339716 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj" pod "6526bc3c-b8e6-4604-b813-6860b62ddb0f" (UID: "6526bc3c-b8e6-4604-b813-6860b62ddb0f"). InnerVolumeSpecName "kube-api-access-4njtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:18.439927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.439816 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njtj\" (UniqueName: \"kubernetes.io/projected/6526bc3c-b8e6-4604-b813-6860b62ddb0f-kube-api-access-4njtj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:18.439927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.439844 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526bc3c-b8e6-4604-b813-6860b62ddb0f-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:18.939878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.939848 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" Oct 31 03:44:18.940125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.939879 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7" event={"ID":"6526bc3c-b8e6-4604-b813-6860b62ddb0f","Type":"ContainerDied","Data":"d43edbf0759ee18239209ebe1054059774705c000a92a4db35e9c7ec343ab685"} Oct 31 03:44:18.940125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:18.939930 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43edbf0759ee18239209ebe1054059774705c000a92a4db35e9c7ec343ab685" Oct 31 03:44:20.308111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:20.304680 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:21.540174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.540134 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:21.850771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.850727 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:21.995423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.995389 4488 generic.go:356] "Generic (PLEG): container finished" podID="789a8caa-b323-4999-b729-3274d34287c5" containerID="2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986" exitCode=0 Oct 31 03:44:21.995868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.995521 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" event={"ID":"789a8caa-b323-4999-b729-3274d34287c5","Type":"ContainerDied","Data":"2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986"} Oct 31 03:44:21.996659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.996637 4488 scope.go:122] "RemoveContainer" containerID="2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986" Oct 31 03:44:21.999494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.999448 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:44:21.999722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.999697 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:44:21.999872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:21.999853 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:44:22.240648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:22.240620 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:22.635013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:22.634575 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx"] Oct 31 03:44:22.640874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:22.640583 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dafd9df5-v78vx"] Oct 31 03:44:22.722525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:22.722001 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km"] Oct 31 03:44:22.722525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:22.722294 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" containerID="cri-o://3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f" gracePeriod=30 Oct 31 03:44:22.740942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:22.740910 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.050574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.045138 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" event={"ID":"789a8caa-b323-4999-b729-3274d34287c5","Type":"ContainerStarted","Data":"bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c"} Oct 31 03:44:23.304628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.304468 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.467546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.467512 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.644114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.643916 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.763113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.752501 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:44:23.763113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.753301 4488 patch_prober.go:29] interesting pod/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:23.763113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.753352 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:23.822118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.816133 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.925336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.925156 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l"] Oct 31 03:44:23.953980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.952226 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e38c94c-7jk4l"] Oct 31 03:44:23.985156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.976804 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8"] Oct 31 03:44:23.985156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:23.977034 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" containerID="cri-o://c2caa3a2b498efc00db1e3742710403a182faa78827ffe0b62eac563a92995b9" gracePeriod=30 Oct 31 03:44:24.030721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.030657 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:24.242213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.239969 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:24.424040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.423477 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:24.659116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.656203 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35127fff-11e4-4ee6-91e1-f818f1577570" path="/var/lib/kubelet/pods/35127fff-11e4-4ee6-91e1-f818f1577570/volumes" Oct 31 03:44:24.659116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.657018 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac" path="/var/lib/kubelet/pods/ae8d9eaa-993e-4ebc-ad59-6af678d3f9ac/volumes" Oct 31 03:44:24.659116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.658165 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:24.747117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.746369 4488 patch_prober.go:29] interesting pod/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:24.747117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:24.746472 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:25.027199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.025677 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:25.052936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.052893 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r"] Oct 31 03:44:25.059491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.059444 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0109f1d-wsv4r"] Oct 31 03:44:25.280177 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.244863 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5"] Oct 31 03:44:25.469564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.469529 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5"] Oct 31 03:44:25.470824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.470801 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6526bc3c-b8e6-4604-b813-6860b62ddb0f" containerName="revoke-keys" Oct 31 03:44:25.470963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.470954 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6526bc3c-b8e6-4604-b813-6860b62ddb0f" containerName="revoke-keys" Oct 31 03:44:25.471193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.471180 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6526bc3c-b8e6-4604-b813-6860b62ddb0f" containerName="revoke-keys" Oct 31 03:44:25.474325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.474306 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.505936 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532316 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532382 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532417 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532649 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532688 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532877 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.532943 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.535066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.533046 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599np\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.555527 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.555491 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-8de398-openshift-default-dockercfg-sgpvs\"" Oct 31 03:44:25.563924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.563882 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5"] Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658351 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658414 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658538 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658582 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658665 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599np\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658793 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658835 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658865 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.658897 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.659553 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.659826 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.662190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.660122 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.694624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.677757 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.757114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.755900 4488 patch_prober.go:29] interesting pod/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:25.757114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.756004 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:25.782122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.772942 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599np\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.794159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.792936 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.798232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.797674 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.798232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.798067 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:25.798484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:25.798272 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy\") pod \"e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:26.090676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.090639 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:26.133168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.131626 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:26.341808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.341728 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d" path="/var/lib/kubelet/pods/cd0c0e32-14f2-42ef-a04c-b6a32a8eb54d/volumes" Oct 31 03:44:26.589025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.588992 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5"] Oct 31 03:44:26.595258 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:44:26.595179 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b4571b_f72d_4116_afb5_f8c63fc23db7.slice/crio-53a1ea92b7ed1d0b34a86b96eba7ff8033899693c4220b343c722cffc8c45e08 WatchSource:0}: Error finding container 53a1ea92b7ed1d0b34a86b96eba7ff8033899693c4220b343c722cffc8c45e08: Status 404 returned error can't find the container with id 53a1ea92b7ed1d0b34a86b96eba7ff8033899693c4220b343c722cffc8c45e08 Oct 31 03:44:26.599583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.599534 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:44:26.599680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.599640 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:44:26.599721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.599687 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:44:26.753551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.752241 4488 patch_prober.go:29] interesting pod/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:26.753551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:26.752324 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:27.060956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.060552 4488 patch_prober.go:29] interesting pod/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:44:27.060956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.060620 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:44:27.199116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.198514 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" event={"ID":"78b4571b-f72d-4116-afb5-f8c63fc23db7","Type":"ContainerStarted","Data":"3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf"} Oct 31 03:44:27.199116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.198587 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" event={"ID":"78b4571b-f72d-4116-afb5-f8c63fc23db7","Type":"ContainerStarted","Data":"53a1ea92b7ed1d0b34a86b96eba7ff8033899693c4220b343c722cffc8c45e08"} Oct 31 03:44:27.436118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.421437 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podStartSLOduration=2.4214169119999998 podStartE2EDuration="2.421416912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:27.366705422 +0000 UTC m=+1495.163423682" watchObservedRunningTime="2026-10-31 03:44:27.421416912 +0000 UTC m=+1495.218135179" Oct 31 03:44:27.743002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.742969 4488 patch_prober.go:29] interesting pod/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:27.743209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:27.743033 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:28.070366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.070339 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:44:28.091753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.091724 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:28.097730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.097699 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104173 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104272 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104352 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104507 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104547 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104651 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104715 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104792 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.109117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.104851 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket\") pod \"d3722cbb-788e-403c-8e29-2430eb5d804b\" (UID: \"d3722cbb-788e-403c-8e29-2430eb5d804b\") " Oct 31 03:44:28.115115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.112393 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:28.115115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.113329 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:28.115115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.113489 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:28.117494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.117461 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:28.129449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.129403 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:28.130698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.130646 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:28.133701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.133662 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "kube-api-access-l7t6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:28.134108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.134057 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:28.134399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.134374 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo" pod "d3722cbb-788e-403c-8e29-2430eb5d804b" (UID: "d3722cbb-788e-403c-8e29-2430eb5d804b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217690 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7t6g\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-kube-api-access-l7t6g\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217735 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217752 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217766 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3722cbb-788e-403c-8e29-2430eb5d804b-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217780 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217794 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217807 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217820 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.226137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.217834 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3722cbb-788e-403c-8e29-2430eb5d804b-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:28.243887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.243854 4488 generic.go:356] "Generic (PLEG): container finished" podID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerID="3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f" exitCode=0 Oct 31 03:44:28.244180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.243936 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" Oct 31 03:44:28.244180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.243957 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" event={"ID":"d3722cbb-788e-403c-8e29-2430eb5d804b","Type":"ContainerDied","Data":"3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f"} Oct 31 03:44:28.244180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.244027 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km" event={"ID":"d3722cbb-788e-403c-8e29-2430eb5d804b","Type":"ContainerDied","Data":"55f3ca18db5cd28776d556875e623a73e14aa1484e45e35a7624568c8802e91e"} Oct 31 03:44:28.244180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.244060 4488 scope.go:122] "RemoveContainer" containerID="3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f" Oct 31 03:44:28.244976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.244953 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:28.246411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.246389 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:44:28.262162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.262139 4488 scope.go:122] "RemoveContainer" containerID="1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297" Oct 31 03:44:28.279448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.279420 4488 scope.go:122] "RemoveContainer" containerID="3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f" Oct 31 03:44:28.279911 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:44:28.279877 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f\": container with ID starting with 3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f not found: ID does not exist" containerID="3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f" Oct 31 03:44:28.280053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.279917 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f"} err="failed to get container status \"3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f\": rpc error: code = NotFound desc = could not find container \"3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f\": container with ID starting with 3994aa823ea135b28f1dd145c724b9bfb0733d55f9371b2b8352f5a6f1e7b58f not found: ID does not exist" Oct 31 03:44:28.280053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.279938 4488 scope.go:122] "RemoveContainer" containerID="1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297" Oct 31 03:44:28.280641 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:44:28.280618 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297\": container with ID starting with 1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297 not found: ID does not exist" containerID="1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297" Oct 31 03:44:28.280735 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.280649 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297"} err="failed to get container status \"1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297\": rpc error: code = NotFound desc = could not find container \"1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297\": container with ID starting with 1e4f7e16cb8e55a1e3f54c98e8c77d65a3b0aebdf2214af800629bf29f6c7297 not found: ID does not exist" Oct 31 03:44:28.319182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.315600 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:28.634928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.634891 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km"] Oct 31 03:44:28.644283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.643151 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dafd9df5-openshift-default-647998c57f-nf6km"] Oct 31 03:44:28.644458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.644307 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:44:28.750550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.746900 4488 patch_prober.go:29] interesting pod/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:28.750550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.746960 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:28.844251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:28.844205 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:29.008486 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.008450 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:29.387308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.384977 4488 generic.go:356] "Generic (PLEG): container finished" podID="8eb7b302-9057-446b-abf7-90c8071fb932" containerID="c2caa3a2b498efc00db1e3742710403a182faa78827ffe0b62eac563a92995b9" exitCode=0 Oct 31 03:44:29.388147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.387966 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" event={"ID":"8eb7b302-9057-446b-abf7-90c8071fb932","Type":"ContainerDied","Data":"c2caa3a2b498efc00db1e3742710403a182faa78827ffe0b62eac563a92995b9"} Oct 31 03:44:29.388147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.388041 4488 scope.go:122] "RemoveContainer" containerID="dd2a444b5219548823a821497f5d2ebeb1583b3001b0e8080e16691f8115f33f" Oct 31 03:44:29.538315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.538285 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:44:29.615383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615348 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.615568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615396 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.615568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615437 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.615568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615481 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.615753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615696 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.615805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615742 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:29.615899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615881 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.615977 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615957 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.616044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.615964 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:29.616044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.616001 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.616044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.616011 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:29.616223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.616064 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket\") pod \"8eb7b302-9057-446b-abf7-90c8071fb932\" (UID: \"8eb7b302-9057-446b-abf7-90c8071fb932\") " Oct 31 03:44:29.616328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.616279 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:29.616436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.616389 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:29.617115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.617061 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.617115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.617108 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.617294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.617136 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb7b302-9057-446b-abf7-90c8071fb932-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.617294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.617149 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.617294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.617163 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.618363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.618337 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:29.618471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.618414 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "kube-api-access-2hkdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:29.618696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.618678 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:29.618759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.618743 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token" pod "8eb7b302-9057-446b-abf7-90c8071fb932" (UID: "8eb7b302-9057-446b-abf7-90c8071fb932"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:29.719660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.719497 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb7b302-9057-446b-abf7-90c8071fb932-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.719660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.719540 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hkdx\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-kube-api-access-2hkdx\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.719660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.719555 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb7b302-9057-446b-abf7-90c8071fb932-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.719660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.719571 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb7b302-9057-446b-abf7-90c8071fb932-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:29.765513 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.765481 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:44:29.767122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:29.767072 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:44:30.283258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.283175 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" podUID="75c2c1d3-6b3a-40be-8127-e8bdaf691104" containerName="payload-pre-processing" containerID="cri-o://9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76" gracePeriod=25 Oct 31 03:44:30.298929 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.298899 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:30.325814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.325775 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" path="/var/lib/kubelet/pods/d3722cbb-788e-403c-8e29-2430eb5d804b/volumes" Oct 31 03:44:30.409511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.409290 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" event={"ID":"8eb7b302-9057-446b-abf7-90c8071fb932","Type":"ContainerDied","Data":"479d9bda8fa75a9dba267dda1c3920950b94480eda86a7b78191b6be90567eee"} Oct 31 03:44:30.409511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.409377 4488 scope.go:122] "RemoveContainer" containerID="c2caa3a2b498efc00db1e3742710403a182faa78827ffe0b62eac563a92995b9" Oct 31 03:44:30.409511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.409397 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8" Oct 31 03:44:30.567403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.567371 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:30.583406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.583376 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8"] Oct 31 03:44:30.589463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.589435 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:44:30.589617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.589482 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9e38c94c-openshift-default-6796d6b985-5gxw8"] Oct 31 03:44:30.736241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.736213 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume\") pod \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " Oct 31 03:44:30.736454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.736434 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt\" (UniqueName: \"kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt\") pod \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\" (UID: \"75c2c1d3-6b3a-40be-8127-e8bdaf691104\") " Oct 31 03:44:30.736590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.736566 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume" pod "75c2c1d3-6b3a-40be-8127-e8bdaf691104" (UID: "75c2c1d3-6b3a-40be-8127-e8bdaf691104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:30.737037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.737014 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c2c1d3-6b3a-40be-8127-e8bdaf691104-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:30.738862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.738840 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt" pod "75c2c1d3-6b3a-40be-8127-e8bdaf691104" (UID: "75c2c1d3-6b3a-40be-8127-e8bdaf691104"). InnerVolumeSpecName "kube-api-access-ts8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:30.839021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.838988 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts8rt\" (UniqueName: \"kubernetes.io/projected/75c2c1d3-6b3a-40be-8127-e8bdaf691104-kube-api-access-ts8rt\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:44:30.929753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:30.929722 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:31.149558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.149467 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r"] Oct 31 03:44:31.154514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.154490 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67b1ef6f-g8f9r"] Oct 31 03:44:31.337061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.337026 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:31.419503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.419389 4488 generic.go:356] "Generic (PLEG): container finished" podID="75c2c1d3-6b3a-40be-8127-e8bdaf691104" containerID="9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76" exitCode=0 Oct 31 03:44:31.419503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.419458 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" Oct 31 03:44:31.420521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.419482 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" event={"ID":"75c2c1d3-6b3a-40be-8127-e8bdaf691104","Type":"ContainerDied","Data":"9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76"} Oct 31 03:44:31.420521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.419601 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5" event={"ID":"75c2c1d3-6b3a-40be-8127-e8bdaf691104","Type":"ContainerDied","Data":"26c505bf53382c477f60a6514f8cd6924d47f091e86e1c4d76e898c738bf3d46"} Oct 31 03:44:31.420521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.419638 4488 scope.go:122] "RemoveContainer" containerID="9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76" Oct 31 03:44:31.448640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.448174 4488 scope.go:122] "RemoveContainer" containerID="9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76" Oct 31 03:44:31.450576 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:44:31.450470 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76\": container with ID starting with 9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76 not found: ID does not exist" containerID="9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76" Oct 31 03:44:31.450576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.450520 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76"} err="failed to get container status \"9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76\": rpc error: code = NotFound desc = could not find container \"9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76\": container with ID starting with 9d1ee34889c16a0441b397d6ccda426b49b07d67a578f7d0cb4383173a4b8c76 not found: ID does not exist" Oct 31 03:44:31.658684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.658631 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:31.678124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.671470 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5"] Oct 31 03:44:31.687143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.686664 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e9fd0a85-79cc74c44c-5hrf5"] Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.984191 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5"] Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985184 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985206 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985233 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985240 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985268 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985275 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985323 4488 cpu_manager.go:414] "containerMap: removing container" podUID="75c2c1d3-6b3a-40be-8127-e8bdaf691104" containerName="payload-pre-processing" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985330 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c2c1d3-6b3a-40be-8127-e8bdaf691104" containerName="payload-pre-processing" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985558 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c2c1d3-6b3a-40be-8127-e8bdaf691104" containerName="payload-pre-processing" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985583 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985596 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985617 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" Oct 31 03:44:31.987769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:31.985631 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" Oct 31 03:44:32.032114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.026398 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5"] Oct 31 03:44:32.032114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.026529 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.085409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.079413 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bb362142\"" Oct 31 03:44:32.085409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.080064 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bb362142-dockercfg-4jkz8\"" Oct 31 03:44:32.174786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.174751 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.174978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.174798 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh8h\" (UniqueName: \"kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h\") pod \"payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.279163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.276635 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.279163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.276698 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh8h\" (UniqueName: \"kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h\") pod \"payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.279163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.278486 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.326830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.326793 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a02fe56-0ef1-4425-bd77-c52f6caf2976" path="/var/lib/kubelet/pods/1a02fe56-0ef1-4425-bd77-c52f6caf2976/volumes" Oct 31 03:44:32.327620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.327596 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c2c1d3-6b3a-40be-8127-e8bdaf691104" path="/var/lib/kubelet/pods/75c2c1d3-6b3a-40be-8127-e8bdaf691104/volumes" Oct 31 03:44:32.328232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.328217 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb7b302-9057-446b-abf7-90c8071fb932" path="/var/lib/kubelet/pods/8eb7b302-9057-446b-abf7-90c8071fb932/volumes" Oct 31 03:44:32.339194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.339167 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh8h\" (UniqueName: \"kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h\") pod \"payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.410211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.410179 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:32.439439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.439411 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bb362142-dockercfg-4jkz8\"" Oct 31 03:44:32.448704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.448678 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:32.594710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.593508 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:32.820197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.820027 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:32.957197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:32.953807 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5"] Oct 31 03:44:33.440863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:33.440821 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" event={"ID":"e0c92668-f7f1-40ca-96d6-f3439daa54b3","Type":"ContainerStarted","Data":"5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059"} Oct 31 03:44:33.441399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:33.440881 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" event={"ID":"e0c92668-f7f1-40ca-96d6-f3439daa54b3","Type":"ContainerStarted","Data":"36ef407ca06be670b4dcf7140eb4e253e8aa3cb48d0e33c6a28c687c610891d5"} Oct 31 03:44:33.441399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:33.440924 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:33.545574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:33.545527 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:33.560630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:33.560565 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" podStartSLOduration=2.560548534 podStartE2EDuration="2.560548534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:33.556808565 +0000 UTC m=+1501.353526826" watchObservedRunningTime="2026-10-31 03:44:33.560548534 +0000 UTC m=+1501.357266787" Oct 31 03:44:33.723195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:33.717135 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:35.820523 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:35.820488 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:36.271355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:36.271321 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:36.514517 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:36.514477 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:36.597001 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:44:36.596960 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:44:39.053395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:39.053364 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm"] Oct 31 03:44:39.103012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:39.102974 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e5f1f43-bvwfm"] Oct 31 03:44:40.318470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:40.318439 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be99c841-f0fd-4ede-a3c5-3a9c60f317c0" path="/var/lib/kubelet/pods/be99c841-f0fd-4ede-a3c5-3a9c60f317c0/volumes" Oct 31 03:44:40.749483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:40.749443 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:41.051034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:41.050946 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj"] Oct 31 03:44:41.100589 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:41.100550 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4688bbf3-phszj"] Oct 31 03:44:42.318801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:42.318766 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6546f9b9-badd-408e-ae87-4fc8299ba7e9" path="/var/lib/kubelet/pods/6546f9b9-badd-408e-ae87-4fc8299ba7e9/volumes" Oct 31 03:44:44.452942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:44.452909 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:44:44.454480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:44.454457 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:45.593586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:45.593548 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:47.585651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:47.585616 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:44:50.063750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:50.063707 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9"] Oct 31 03:44:50.101318 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:50.101280 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb037eba-2ndd9"] Oct 31 03:44:50.324900 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:50.324790 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fbda48-9568-431a-a8a1-86da1c8b791b" path="/var/lib/kubelet/pods/d7fbda48-9568-431a-a8a1-86da1c8b791b/volumes" Oct 31 03:44:50.532443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:50.532406 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:51.079775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:51.079740 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb"] Oct 31 03:44:51.087294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:51.087257 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b265bdb-h2mkb"] Oct 31 03:44:52.318860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:52.318829 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f514e46-ed61-49f9-93ac-19e73cb21544" path="/var/lib/kubelet/pods/5f514e46-ed61-49f9-93ac-19e73cb21544/volumes" Oct 31 03:44:55.302241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:55.302212 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:56.052536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:56.052503 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b"] Oct 31 03:44:56.102321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:56.102285 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7edba4ad-nsg8b"] Oct 31 03:44:56.318649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:56.318562 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ae3bea-90ba-4e93-b928-9ab3fb43b855" path="/var/lib/kubelet/pods/13ae3bea-90ba-4e93-b928-9ab3fb43b855/volumes" Oct 31 03:44:59.480240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:44:59.480193 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:00.173075 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.173037 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g"] Oct 31 03:45:00.174747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.174719 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" Oct 31 03:45:00.174747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.174746 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3722cbb-788e-403c-8e29-2430eb5d804b" containerName="istio-proxy" Oct 31 03:45:00.184468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.184425 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.237772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.237732 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:45:00.240443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.240421 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-476rz\"" Oct 31 03:45:00.293435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.293391 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4"] Oct 31 03:45:00.297779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.297754 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.302972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.302944 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w"] Oct 31 03:45:00.349123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.338744 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.349123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.338964 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.349123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.339066 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.366121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.362790 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr"] Oct 31 03:45:00.387148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.382434 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:45:00.387148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.382507 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:45:00.387148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.382550 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:45:00.387148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.382646 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:45:00.387148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.382729 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:45:00.409138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.389335 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.521111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.505882 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.577515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.577474 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.577824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.519446 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.578739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.519345 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv"] Oct 31 03:45:00.582080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.578070 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv28q\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.582080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.581872 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.595036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.589336 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g"] Oct 31 03:45:00.595036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.590449 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.611321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.611243 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d"] Oct 31 03:45:00.615722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.615698 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4"] Oct 31 03:45:00.615851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.615776 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr"] Oct 31 03:45:00.615851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.615804 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w"] Oct 31 03:45:00.615851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.615828 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv"] Oct 31 03:45:00.615973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.615852 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d"] Oct 31 03:45:00.615973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.615962 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.683769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.683723 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwr4\" (UniqueName: \"kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.684002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.683791 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.684002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.683905 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.684214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684025 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gssxd\" (UniqueName: \"kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.684214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684069 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.684214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684144 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.684385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684204 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.684385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684255 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.684385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684315 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzzf\" (UniqueName: \"kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf\") pod \"maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.684549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684414 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.684549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684518 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9d6\" (UniqueName: \"kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6\") pod \"maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.684662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684609 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv28q\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.684816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684792 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.685071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684835 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.685071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.684889 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.685235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.685156 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tz8\" (UniqueName: \"kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8\") pod \"maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.685235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.685227 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.685766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.685739 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.686915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.686891 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.687301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.687282 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.687725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.687708 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.688177 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.687803 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.752747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.752674 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv28q\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.752747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.752718 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6c56g\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.786868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.786835 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.787065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.786902 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tz8\" (UniqueName: \"kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8\") pod \"maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.787065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.786942 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.787065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787008 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwr4\" (UniqueName: \"kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.787065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787029 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.787320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787211 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssxd\" (UniqueName: \"kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.787320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787264 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.787422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787340 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzzf\" (UniqueName: \"kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf\") pod \"maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.787422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787367 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.787531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787429 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9d6\" (UniqueName: \"kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6\") pod \"maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.787634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787611 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.787701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787657 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.787794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.787773 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.788174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.788158 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.788325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.788303 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.798785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.798756 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:00.849992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.849956 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tz8\" (UniqueName: \"kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8\") pod \"maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.850204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.849991 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzzf\" (UniqueName: \"kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf\") pod \"maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.850204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.850029 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9d6\" (UniqueName: \"kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6\") pod \"maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:00.850586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.850563 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssxd\" (UniqueName: \"kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:00.852352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.852295 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwr4\" (UniqueName: \"kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:00.909540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.909503 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:00.918763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.918736 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:00.929280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:00.929250 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:01.106200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.106165 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:01.110427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.110397 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:01.130169 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.130121 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:01.201414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.201385 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g"] Oct 31 03:45:01.205553 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:01.205477 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491878a4_0f7d_4970_8ebc_bffb94fcef66.slice/crio-a4f08c8c9a02962c5bdf27da51364f82952434ec2e5495bba18fa75328a068b2 WatchSource:0}: Error finding container a4f08c8c9a02962c5bdf27da51364f82952434ec2e5495bba18fa75328a068b2: Status 404 returned error can't find the container with id a4f08c8c9a02962c5bdf27da51364f82952434ec2e5495bba18fa75328a068b2 Oct 31 03:45:01.214620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.208741 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:45:01.337414 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:01.337386 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f76e8f2_d978_41d3_85d7_df75b8611aeb.slice/crio-bf8df7c0c9597ad7d17f00a0ad999edf30d3f94e4dd025ba34c836c0c74f78f6 WatchSource:0}: Error finding container bf8df7c0c9597ad7d17f00a0ad999edf30d3f94e4dd025ba34c836c0c74f78f6: Status 404 returned error can't find the container with id bf8df7c0c9597ad7d17f00a0ad999edf30d3f94e4dd025ba34c836c0c74f78f6 Oct 31 03:45:01.339116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.339077 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv"] Oct 31 03:45:01.354325 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:01.354300 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47656ece_d061_494c_9676_a31beceda2f5.slice/crio-dcc345c88be3ddc7ed07b2f71c97369c40f152df553ece3a22051e1b29f9a16e WatchSource:0}: Error finding container dcc345c88be3ddc7ed07b2f71c97369c40f152df553ece3a22051e1b29f9a16e: Status 404 returned error can't find the container with id dcc345c88be3ddc7ed07b2f71c97369c40f152df553ece3a22051e1b29f9a16e Oct 31 03:45:01.355492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.355462 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr"] Oct 31 03:45:01.360312 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:01.360286 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361dffd6_4017_4026_b972_c04f9c49ce1b.slice/crio-2227153f8292f15aa40967d203ca31d419b4dc9acbe31b7775e056428e3b37c2 WatchSource:0}: Error finding container 2227153f8292f15aa40967d203ca31d419b4dc9acbe31b7775e056428e3b37c2: Status 404 returned error can't find the container with id 2227153f8292f15aa40967d203ca31d419b4dc9acbe31b7775e056428e3b37c2 Oct 31 03:45:01.363535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.363506 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d"] Oct 31 03:45:01.401386 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.401358 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:01.512432 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:01.512376 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f31640_b60c_4ffa_9a19_f8ad3d84638a.slice/crio-baf46551aa2cf5957bbf1b62c9711db54b596ffba61d243a30b1a750001ceb06 WatchSource:0}: Error finding container baf46551aa2cf5957bbf1b62c9711db54b596ffba61d243a30b1a750001ceb06: Status 404 returned error can't find the container with id baf46551aa2cf5957bbf1b62c9711db54b596ffba61d243a30b1a750001ceb06 Oct 31 03:45:01.524508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.523187 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4"] Oct 31 03:45:01.552978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.552923 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w"] Oct 31 03:45:01.566768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.566729 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:01.628502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.628472 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7"] Oct 31 03:45:01.633407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.633382 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9fd0a85-zqmq7"] Oct 31 03:45:01.784648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.784607 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerStarted","Data":"d62ce00e4f3249cacb28be9758ef1292b3056a0af6227a7a719144a13bbac137"} Oct 31 03:45:01.786308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.786281 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerStarted","Data":"2227153f8292f15aa40967d203ca31d419b4dc9acbe31b7775e056428e3b37c2"} Oct 31 03:45:01.787704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.787681 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerStarted","Data":"dcc345c88be3ddc7ed07b2f71c97369c40f152df553ece3a22051e1b29f9a16e"} Oct 31 03:45:01.789214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.789194 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" event={"ID":"491878a4-0f7d-4970-8ebc-bffb94fcef66","Type":"ContainerStarted","Data":"a4f08c8c9a02962c5bdf27da51364f82952434ec2e5495bba18fa75328a068b2"} Oct 31 03:45:01.790565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.790543 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerStarted","Data":"bf8df7c0c9597ad7d17f00a0ad999edf30d3f94e4dd025ba34c836c0c74f78f6"} Oct 31 03:45:01.791726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:01.791704 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerStarted","Data":"baf46551aa2cf5957bbf1b62c9711db54b596ffba61d243a30b1a750001ceb06"} Oct 31 03:45:02.166628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.166593 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr"] Oct 31 03:45:02.166888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.166860 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" containerID="cri-o://bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c" gracePeriod=30 Oct 31 03:45:02.218447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.218415 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:02.411545 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.390820 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6526bc3c-b8e6-4604-b813-6860b62ddb0f" path="/var/lib/kubelet/pods/6526bc3c-b8e6-4604-b813-6860b62ddb0f/volumes" Oct 31 03:45:02.802833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.802802 4488 generic.go:356] "Generic (PLEG): container finished" podID="47656ece-d061-494c-9676-a31beceda2f5" containerID="79ea9d637a078a213d357a6bae1dc3fe2a5f8a8594ecac8f03d6039b5cb5e773" exitCode=60 Oct 31 03:45:02.803332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.802889 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerDied","Data":"79ea9d637a078a213d357a6bae1dc3fe2a5f8a8594ecac8f03d6039b5cb5e773"} Oct 31 03:45:02.803332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.803278 4488 scope.go:122] "RemoveContainer" containerID="79ea9d637a078a213d357a6bae1dc3fe2a5f8a8594ecac8f03d6039b5cb5e773" Oct 31 03:45:02.804798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.804771 4488 generic.go:356] "Generic (PLEG): container finished" podID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerID="ececd4e15c29022c0762fcfe374b85883ef285be936ee24f77be7afd192e8edf" exitCode=60 Oct 31 03:45:02.804932 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.804865 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerDied","Data":"ececd4e15c29022c0762fcfe374b85883ef285be936ee24f77be7afd192e8edf"} Oct 31 03:45:02.805187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.805165 4488 scope.go:122] "RemoveContainer" containerID="ececd4e15c29022c0762fcfe374b85883ef285be936ee24f77be7afd192e8edf" Oct 31 03:45:02.807061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.807017 4488 generic.go:356] "Generic (PLEG): container finished" podID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerID="216cafb3fe40c55d8dfc22bc8e7e5a16fb508eebfdb2545fefc6df6efe5b35b9" exitCode=60 Oct 31 03:45:02.807164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.807123 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerDied","Data":"216cafb3fe40c55d8dfc22bc8e7e5a16fb508eebfdb2545fefc6df6efe5b35b9"} Oct 31 03:45:02.807368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.807352 4488 scope.go:122] "RemoveContainer" containerID="216cafb3fe40c55d8dfc22bc8e7e5a16fb508eebfdb2545fefc6df6efe5b35b9" Oct 31 03:45:02.810952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.810932 4488 generic.go:356] "Generic (PLEG): container finished" podID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerID="a3d580374d858becb9ad5007e0f09a3c70eb1345214ab3a9f5cdee3022f77577" exitCode=60 Oct 31 03:45:02.811048 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.810988 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerDied","Data":"a3d580374d858becb9ad5007e0f09a3c70eb1345214ab3a9f5cdee3022f77577"} Oct 31 03:45:02.811339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.811320 4488 scope.go:122] "RemoveContainer" containerID="a3d580374d858becb9ad5007e0f09a3c70eb1345214ab3a9f5cdee3022f77577" Oct 31 03:45:02.820323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.820292 4488 generic.go:356] "Generic (PLEG): container finished" podID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerID="7fd993be5f667c1eefe390077aeb743226f156f9e88128c102c3cc564d31c53b" exitCode=60 Oct 31 03:45:02.820533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.820331 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerDied","Data":"7fd993be5f667c1eefe390077aeb743226f156f9e88128c102c3cc564d31c53b"} Oct 31 03:45:02.820728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:02.820698 4488 scope.go:122] "RemoveContainer" containerID="7fd993be5f667c1eefe390077aeb743226f156f9e88128c102c3cc564d31c53b" Oct 31 03:45:03.834869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:03.834832 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerStarted","Data":"8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047"} Oct 31 03:45:03.837012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:03.836982 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerStarted","Data":"d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3"} Oct 31 03:45:03.839150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:03.839120 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerStarted","Data":"9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa"} Oct 31 03:45:03.841415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:03.841343 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerStarted","Data":"9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f"} Oct 31 03:45:03.845864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:03.845840 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerStarted","Data":"d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af"} Oct 31 03:45:03.952408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:03.952348 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" podStartSLOduration=3.952332154 podStartE2EDuration="3.952332154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:03.952060313 +0000 UTC m=+1531.748778573" watchObservedRunningTime="2026-10-31 03:45:03.952332154 +0000 UTC m=+1531.749050416" Oct 31 03:45:04.082107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.082028 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" podStartSLOduration=4.082008532 podStartE2EDuration="4.082008532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:04.068325048 +0000 UTC m=+1531.865043308" watchObservedRunningTime="2026-10-31 03:45:04.082008532 +0000 UTC m=+1531.878726791" Oct 31 03:45:04.175829 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:04.175796 4488 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3186ddc_193c_42c7_9122_b61afd9f7944.slice/crio-d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f31640_b60c_4ffa_9a19_f8ad3d84638a.slice/crio-conmon-9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:45:04.186414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.186359 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" podStartSLOduration=4.186341997 podStartE2EDuration="4.186341997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:04.183530372 +0000 UTC m=+1531.980248662" watchObservedRunningTime="2026-10-31 03:45:04.186341997 +0000 UTC m=+1531.983060256" Oct 31 03:45:04.305564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.305499 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" podStartSLOduration=4.305477207 podStartE2EDuration="4.305477207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:04.298525653 +0000 UTC m=+1532.095243911" watchObservedRunningTime="2026-10-31 03:45:04.305477207 +0000 UTC m=+1532.102195466" Oct 31 03:45:04.414890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.414791 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" podStartSLOduration=4.414776531 podStartE2EDuration="4.414776531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:04.413486324 +0000 UTC m=+1532.210204583" watchObservedRunningTime="2026-10-31 03:45:04.414776531 +0000 UTC m=+1532.211494837" Oct 31 03:45:04.859922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.859877 4488 generic.go:356] "Generic (PLEG): container finished" podID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerID="8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047" exitCode=60 Oct 31 03:45:04.860501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.860052 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerDied","Data":"8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047"} Oct 31 03:45:04.860501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.860160 4488 scope.go:122] "RemoveContainer" containerID="7fd993be5f667c1eefe390077aeb743226f156f9e88128c102c3cc564d31c53b" Oct 31 03:45:04.860501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.860444 4488 scope.go:122] "RemoveContainer" containerID="8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047" Oct 31 03:45:04.860910 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:04.860881 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d_odh-ai-gateway-infra(361dffd6-4017-4026-b972-c04f9c49ce1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" Oct 31 03:45:04.863338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.863316 4488 generic.go:356] "Generic (PLEG): container finished" podID="47656ece-d061-494c-9676-a31beceda2f5" containerID="d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3" exitCode=60 Oct 31 03:45:04.863445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.863401 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerDied","Data":"d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3"} Oct 31 03:45:04.863766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.863748 4488 scope.go:122] "RemoveContainer" containerID="d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3" Oct 31 03:45:04.864270 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:04.864235 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr_odh-ai-gateway-infra(47656ece-d061-494c-9676-a31beceda2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" podUID="47656ece-d061-494c-9676-a31beceda2f5" Oct 31 03:45:04.866243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.866216 4488 generic.go:356] "Generic (PLEG): container finished" podID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerID="9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa" exitCode=60 Oct 31 03:45:04.866403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.866371 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerDied","Data":"9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa"} Oct 31 03:45:04.866979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.866956 4488 scope.go:122] "RemoveContainer" containerID="9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa" Oct 31 03:45:04.867495 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:04.867457 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv_odh-ai-gateway-infra(6f76e8f2-d978-41d3-85d7-df75b8611aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" Oct 31 03:45:04.869632 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.869609 4488 generic.go:356] "Generic (PLEG): container finished" podID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerID="9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f" exitCode=60 Oct 31 03:45:04.869733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.869693 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerDied","Data":"9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f"} Oct 31 03:45:04.869989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.869972 4488 scope.go:122] "RemoveContainer" containerID="9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f" Oct 31 03:45:04.870439 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:04.870419 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4_odh-ai-gateway-infra(03f31640-b60c-4ffa-9a19-f8ad3d84638a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" Oct 31 03:45:04.872621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.872535 4488 generic.go:356] "Generic (PLEG): container finished" podID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerID="d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af" exitCode=60 Oct 31 03:45:04.872621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.872577 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerDied","Data":"d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af"} Oct 31 03:45:04.872954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.872936 4488 scope.go:122] "RemoveContainer" containerID="d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af" Oct 31 03:45:04.873393 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:04.873355 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w_odh-ai-gateway-infra(b3186ddc-193c-42c7-9122-b61afd9f7944)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" Oct 31 03:45:04.879891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.879868 4488 scope.go:122] "RemoveContainer" containerID="79ea9d637a078a213d357a6bae1dc3fe2a5f8a8594ecac8f03d6039b5cb5e773" Oct 31 03:45:04.894713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.894688 4488 scope.go:122] "RemoveContainer" containerID="ececd4e15c29022c0762fcfe374b85883ef285be936ee24f77be7afd192e8edf" Oct 31 03:45:04.913424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.913403 4488 scope.go:122] "RemoveContainer" containerID="216cafb3fe40c55d8dfc22bc8e7e5a16fb508eebfdb2545fefc6df6efe5b35b9" Oct 31 03:45:04.929002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:04.928974 4488 scope.go:122] "RemoveContainer" containerID="a3d580374d858becb9ad5007e0f09a3c70eb1345214ab3a9f5cdee3022f77577" Oct 31 03:45:05.181497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.181459 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:05.181634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.181536 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:05.884431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.884403 4488 scope.go:122] "RemoveContainer" containerID="d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3" Oct 31 03:45:05.885014 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:05.884847 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr_odh-ai-gateway-infra(47656ece-d061-494c-9676-a31beceda2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" podUID="47656ece-d061-494c-9676-a31beceda2f5" Oct 31 03:45:05.886178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.886141 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" event={"ID":"491878a4-0f7d-4970-8ebc-bffb94fcef66","Type":"ContainerStarted","Data":"41d21f04e8725cae44b29126c7349bff709dd09c1ee78dceb9bff523d43951d7"} Oct 31 03:45:05.886328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.886295 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:05.888420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.888395 4488 scope.go:122] "RemoveContainer" containerID="9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa" Oct 31 03:45:05.888861 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:05.888829 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv_odh-ai-gateway-infra(6f76e8f2-d978-41d3-85d7-df75b8611aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" Oct 31 03:45:05.890310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.890290 4488 scope.go:122] "RemoveContainer" containerID="9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f" Oct 31 03:45:05.890663 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:05.890640 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4_odh-ai-gateway-infra(03f31640-b60c-4ffa-9a19-f8ad3d84638a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" Oct 31 03:45:05.892135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.892117 4488 scope.go:122] "RemoveContainer" containerID="d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af" Oct 31 03:45:05.892491 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:05.892471 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w_odh-ai-gateway-infra(b3186ddc-193c-42c7-9122-b61afd9f7944)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" Oct 31 03:45:05.893837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:05.893819 4488 scope.go:122] "RemoveContainer" containerID="8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047" Oct 31 03:45:05.894183 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:05.894164 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d_odh-ai-gateway-infra(361dffd6-4017-4026-b972-c04f9c49ce1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" Oct 31 03:45:06.357882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:06.357824 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" podStartSLOduration=2.3869633439999998 podStartE2EDuration="6.357809369s" totalImagesPullingTime="3.970846025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:06.355161105 +0000 UTC m=+1534.151879358" watchObservedRunningTime="2026-10-31 03:45:06.357809369 +0000 UTC m=+1534.154527627" Oct 31 03:45:06.510691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:06.510654 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:06.663387 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:06.663297 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:45:06.907249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:06.907195 4488 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6c56g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:45:06.907652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:06.907262 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:45:07.465937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.465865 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:45:07.554274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554238 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554288 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554321 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554415 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554509 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:45:07.554656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554540 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554609 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554672 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554728 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554795 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket\") pod \"789a8caa-b323-4999-b729-3274d34287c5\" (UID: \"789a8caa-b323-4999-b729-3274d34287c5\") " Oct 31 03:45:07.554965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554891 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:45:07.555022 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.554966 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:45:07.555315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555188 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:45:07.555315 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555227 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:07.555927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555892 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.556042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555932 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.556042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555964 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789a8caa-b323-4999-b729-3274d34287c5-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.556042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555979 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.556042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.555993 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.556978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.556953 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:45:07.557074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.557000 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "kube-api-access-rwnc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:07.557185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.557135 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:45:07.557333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.557310 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token" pod "789a8caa-b323-4999-b729-3274d34287c5" (UID: "789a8caa-b323-4999-b729-3274d34287c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:07.658455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.658414 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.658455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.658461 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnc4\" (UniqueName: \"kubernetes.io/projected/789a8caa-b323-4999-b729-3274d34287c5-kube-api-access-rwnc4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.658690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.658480 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789a8caa-b323-4999-b729-3274d34287c5-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.658690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.658494 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789a8caa-b323-4999-b729-3274d34287c5-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:07.916079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.916043 4488 generic.go:356] "Generic (PLEG): container finished" podID="789a8caa-b323-4999-b729-3274d34287c5" containerID="bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c" exitCode=0 Oct 31 03:45:07.917274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.916114 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" Oct 31 03:45:07.917274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.916116 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" event={"ID":"789a8caa-b323-4999-b729-3274d34287c5","Type":"ContainerDied","Data":"bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c"} Oct 31 03:45:07.917274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.916172 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr" event={"ID":"789a8caa-b323-4999-b729-3274d34287c5","Type":"ContainerDied","Data":"6b4ec2da287d8068bcb38f1f9dd51b8418489a80746cdd7ffb034d8c033e75f2"} Oct 31 03:45:07.917274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.916195 4488 scope.go:122] "RemoveContainer" containerID="bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c" Oct 31 03:45:07.929249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.929223 4488 scope.go:122] "RemoveContainer" containerID="2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986" Oct 31 03:45:07.944039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.944018 4488 scope.go:122] "RemoveContainer" containerID="bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c" Oct 31 03:45:07.944428 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:07.944407 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c\": container with ID starting with bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c not found: ID does not exist" containerID="bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c" Oct 31 03:45:07.944505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.944441 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c"} err="failed to get container status \"bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c\": rpc error: code = NotFound desc = could not find container \"bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c\": container with ID starting with bec402e560d0dd7c6787c5fda7361d222eda510c3c8f798363764f2d054b7f3c not found: ID does not exist" Oct 31 03:45:07.944505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.944468 4488 scope.go:122] "RemoveContainer" containerID="2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986" Oct 31 03:45:07.944742 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:07.944722 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986\": container with ID starting with 2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986 not found: ID does not exist" containerID="2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986" Oct 31 03:45:07.944798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:07.944748 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986"} err="failed to get container status \"2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986\": rpc error: code = NotFound desc = could not find container \"2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986\": container with ID starting with 2a84e1ef803e1e1ddad78579fbe343177169db8e1853bccae43dc73ed35ee986 not found: ID does not exist" Oct 31 03:45:08.051234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:08.051203 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn"] Oct 31 03:45:08.056225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:08.056192 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f74683d-qpxsn"] Oct 31 03:45:08.085491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:08.085457 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr"] Oct 31 03:45:08.101554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:08.101522 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e9fd0a85-openshift-default-7b77bd8989-gz2pr"] Oct 31 03:45:08.318525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:08.318438 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789a8caa-b323-4999-b729-3274d34287c5" path="/var/lib/kubelet/pods/789a8caa-b323-4999-b729-3274d34287c5/volumes" Oct 31 03:45:08.318954 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:08.318939 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8708546d-ea7d-470c-a6af-1a17a5376977" path="/var/lib/kubelet/pods/8708546d-ea7d-470c-a6af-1a17a5376977/volumes" Oct 31 03:45:09.905816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:09.905780 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 03:45:11.422744 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:11.422712 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:13.795489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:13.795449 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:14.313806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.313769 4488 scope.go:122] "RemoveContainer" containerID="d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3" Oct 31 03:45:14.685015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.684974 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd"] Oct 31 03:45:14.685991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.685968 4488 cpu_manager.go:414] "containerMap: removing container" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" Oct 31 03:45:14.686118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.685994 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" Oct 31 03:45:14.686118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.686028 4488 cpu_manager.go:414] "containerMap: removing container" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" Oct 31 03:45:14.686118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.686035 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" Oct 31 03:45:14.686279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.686221 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" Oct 31 03:45:14.695710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.695684 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.779979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.779924 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780054 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780204 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780314 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780421 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4m9\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780471 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780499 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780575 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.780723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.780666 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.782481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.782450 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:14.792140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.792107 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd"] Oct 31 03:45:14.848558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.844943 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e38e21-openshift-default-dockercfg-5d45j\"" Oct 31 03:45:14.882724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882620 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4m9\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.882724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882704 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.882973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882745 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.882973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882784 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.882973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882848 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.882973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882897 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.882973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.882958 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.883274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.883105 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.883274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.883232 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.883713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.883688 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.886163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.884624 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.886163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.885386 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.886163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.885622 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.886459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.886419 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.889623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.889594 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.892380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.892353 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.945259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.945217 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.945259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.945241 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4m9\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9\") pod \"e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:14.972074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.972035 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:14.991131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:14.990080 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerStarted","Data":"254cd39f7e969c0ec7af2dcb471e64fa705260481d02f7aebddcdefcefb77270"} Oct 31 03:45:15.011043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.011012 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:15.167319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.166814 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674"] Oct 31 03:45:15.169166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.167614 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="789a8caa-b323-4999-b729-3274d34287c5" containerName="istio-proxy" Oct 31 03:45:15.173525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.173503 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.242175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.242136 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674"] Oct 31 03:45:15.291005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.290969 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf69280-a762-4868-9057-76241c023adf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291019 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291132 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkww\" (UniqueName: \"kubernetes.io/projected/eaf69280-a762-4868-9057-76241c023adf-kube-api-access-8tkww\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291235 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf69280-a762-4868-9057-76241c023adf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291259 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf69280-a762-4868-9057-76241c023adf-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291314 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291464 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291519 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.291638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.291594 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.310584 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.310551 4488 scope.go:122] "RemoveContainer" containerID="d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af" Oct 31 03:45:15.310799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.310778 4488 scope.go:122] "RemoveContainer" containerID="9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa" Oct 31 03:45:15.393101 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393056 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf69280-a762-4868-9057-76241c023adf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393146 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf69280-a762-4868-9057-76241c023adf-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393197 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393390 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393557 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393517 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393740 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393785 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393735 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393919 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf69280-a762-4868-9057-76241c023adf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.393944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.393927 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.394057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.394003 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.394134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.394058 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkww\" (UniqueName: \"kubernetes.io/projected/eaf69280-a762-4868-9057-76241c023adf-kube-api-access-8tkww\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.394540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.394516 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf69280-a762-4868-9057-76241c023adf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.394642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.394551 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.396163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.396143 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf69280-a762-4868-9057-76241c023adf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.396254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.396200 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf69280-a762-4868-9057-76241c023adf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.467830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.467799 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd"] Oct 31 03:45:15.505298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.505265 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf69280-a762-4868-9057-76241c023adf-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.505535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.505512 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkww\" (UniqueName: \"kubernetes.io/projected/eaf69280-a762-4868-9057-76241c023adf-kube-api-access-8tkww\") pod \"maas-default-gateway-openshift-default-fd4b99879-6d674\" (UID: \"eaf69280-a762-4868-9057-76241c023adf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:15.508120 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:15.508080 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf94248_c7de_465f_9f12_f9a0275f5a5d.slice/crio-9926910c045b5c12373b716e172ae7c423ab301fdf768bc7d7687fd09a65de58 WatchSource:0}: Error finding container 9926910c045b5c12373b716e172ae7c423ab301fdf768bc7d7687fd09a65de58: Status 404 returned error can't find the container with id 9926910c045b5c12373b716e172ae7c423ab301fdf768bc7d7687fd09a65de58 Oct 31 03:45:15.511730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.511691 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:15.511843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.511770 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:15.511843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.511808 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:15.792708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:15.792674 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:16.003370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.003276 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerStarted","Data":"e17b500cae316336a5151a24ca00a5045671ac887a26ca3ddd5e12207eadd1fd"} Oct 31 03:45:16.007296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.007265 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" event={"ID":"ccf94248-c7de-465f-9f12-f9a0275f5a5d","Type":"ContainerStarted","Data":"14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a"} Oct 31 03:45:16.007465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.007324 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" event={"ID":"ccf94248-c7de-465f-9f12-f9a0275f5a5d","Type":"ContainerStarted","Data":"9926910c045b5c12373b716e172ae7c423ab301fdf768bc7d7687fd09a65de58"} Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.014962 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.016069 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerStarted","Data":"eb6707b99fc70973186d7b3267cfa4f97bbfda3e9704625915ef10bf0198fad9"} Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.016554 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.016609 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.018785 4488 generic.go:356] "Generic (PLEG): container finished" podID="47656ece-d061-494c-9676-a31beceda2f5" containerID="254cd39f7e969c0ec7af2dcb471e64fa705260481d02f7aebddcdefcefb77270" exitCode=60 Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.018821 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerDied","Data":"254cd39f7e969c0ec7af2dcb471e64fa705260481d02f7aebddcdefcefb77270"} Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.018870 4488 scope.go:122] "RemoveContainer" containerID="d3cef40e9d62190d6029441f2f5c2c61a4d0e1825df0c92f64ba2c24f6b535b3" Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.019230 4488 scope.go:122] "RemoveContainer" containerID="254cd39f7e969c0ec7af2dcb471e64fa705260481d02f7aebddcdefcefb77270" Oct 31 03:45:16.023138 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:16.019578 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr_odh-ai-gateway-infra(47656ece-d061-494c-9676-a31beceda2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" podUID="47656ece-d061-494c-9676-a31beceda2f5" Oct 31 03:45:16.055420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.055374 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd"] Oct 31 03:45:16.063292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.063259 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fa3641-5zdwd"] Oct 31 03:45:16.144195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.144151 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr"] Oct 31 03:45:16.243550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.243510 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:16.244798 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:16.244766 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf69280_a762_4868_9057_76241c023adf.slice/crio-2c20421ffd18cfe5c640358c35531c523d374596739712e54af62d1bb0d20d1b WatchSource:0}: Error finding container 2c20421ffd18cfe5c640358c35531c523d374596739712e54af62d1bb0d20d1b: Status 404 returned error can't find the container with id 2c20421ffd18cfe5c640358c35531c523d374596739712e54af62d1bb0d20d1b Oct 31 03:45:16.245523 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.245502 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674"] Oct 31 03:45:16.247677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.247641 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:16.247794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.247724 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:16.247794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.247766 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:16.249668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.249380 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podStartSLOduration=2.249363267 podStartE2EDuration="2.249363267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:16.241455772 +0000 UTC m=+1544.038174034" watchObservedRunningTime="2026-10-31 03:45:16.249363267 +0000 UTC m=+1544.046081527" Oct 31 03:45:16.320005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.319979 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a119fa-5656-410b-94eb-f8becd5a1df6" path="/var/lib/kubelet/pods/c5a119fa-5656-410b-94eb-f8becd5a1df6/volumes" Oct 31 03:45:16.395474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.395439 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:16.551357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:16.551257 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:17.016566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.016538 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:17.029252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.029216 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" event={"ID":"eaf69280-a762-4868-9057-76241c023adf","Type":"ContainerStarted","Data":"c2bed18b86580705a7c1234a5cd609ca115a07a42e612e472b4b72ba3eb5de0d"} Oct 31 03:45:17.029387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.029280 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" event={"ID":"eaf69280-a762-4868-9057-76241c023adf","Type":"ContainerStarted","Data":"2c20421ffd18cfe5c640358c35531c523d374596739712e54af62d1bb0d20d1b"} Oct 31 03:45:17.033298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.033272 4488 generic.go:356] "Generic (PLEG): container finished" podID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerID="e17b500cae316336a5151a24ca00a5045671ac887a26ca3ddd5e12207eadd1fd" exitCode=60 Oct 31 03:45:17.033428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.033353 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerDied","Data":"e17b500cae316336a5151a24ca00a5045671ac887a26ca3ddd5e12207eadd1fd"} Oct 31 03:45:17.033481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.033428 4488 scope.go:122] "RemoveContainer" containerID="9bcdd8e177cd292ea68901ff77020a17c04176102dc0ebd066eb969576ed82aa" Oct 31 03:45:17.033864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.033844 4488 scope.go:122] "RemoveContainer" containerID="e17b500cae316336a5151a24ca00a5045671ac887a26ca3ddd5e12207eadd1fd" Oct 31 03:45:17.034279 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:17.034254 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv_odh-ai-gateway-infra(6f76e8f2-d978-41d3-85d7-df75b8611aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" Oct 31 03:45:17.036228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.036202 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerDied","Data":"eb6707b99fc70973186d7b3267cfa4f97bbfda3e9704625915ef10bf0198fad9"} Oct 31 03:45:17.036370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.036281 4488 generic.go:356] "Generic (PLEG): container finished" podID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerID="eb6707b99fc70973186d7b3267cfa4f97bbfda3e9704625915ef10bf0198fad9" exitCode=60 Oct 31 03:45:17.036681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.036662 4488 scope.go:122] "RemoveContainer" containerID="eb6707b99fc70973186d7b3267cfa4f97bbfda3e9704625915ef10bf0198fad9" Oct 31 03:45:17.036987 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:17.036963 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w_odh-ai-gateway-infra(b3186ddc-193c-42c7-9122-b61afd9f7944)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" Oct 31 03:45:17.037070 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.037014 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:17.037720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.037700 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:45:17.050213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.050191 4488 scope.go:122] "RemoveContainer" containerID="d6540fcac2fa52f0c5f3570015b713e445b186ac6795933ad8387523fa0f20af" Oct 31 03:45:17.139001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.138971 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv"] Oct 31 03:45:17.220536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.220507 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:17.309453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.309377 4488 scope.go:122] "RemoveContainer" containerID="9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f" Oct 31 03:45:17.327651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.327606 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca\") pod \"47656ece-d061-494c-9676-a31beceda2f5\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " Oct 31 03:45:17.327741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.327726 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8\" (UniqueName: \"kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8\") pod \"47656ece-d061-494c-9676-a31beceda2f5\" (UID: \"47656ece-d061-494c-9676-a31beceda2f5\") " Oct 31 03:45:17.327987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.327964 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca" pod "47656ece-d061-494c-9676-a31beceda2f5" (UID: "47656ece-d061-494c-9676-a31beceda2f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:17.328554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.328531 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47656ece-d061-494c-9676-a31beceda2f5-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:17.330060 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.330036 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8" pod "47656ece-d061-494c-9676-a31beceda2f5" (UID: "47656ece-d061-494c-9676-a31beceda2f5"). InnerVolumeSpecName "kube-api-access-25tz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:17.372752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.372716 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w"] Oct 31 03:45:17.386663 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.386599 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podStartSLOduration=2.386583951 podStartE2EDuration="2.386583951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:17.380320584 +0000 UTC m=+1545.177038882" watchObservedRunningTime="2026-10-31 03:45:17.386583951 +0000 UTC m=+1545.183302209" Oct 31 03:45:17.435940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.430926 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tz8\" (UniqueName: \"kubernetes.io/projected/47656ece-d061-494c-9676-a31beceda2f5-kube-api-access-25tz8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:17.794441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.794409 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:17.795809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.795779 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:17.796039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:17.795834 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:18.048615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.048521 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerStarted","Data":"1466f008c3fad064a6ac2789e6e59061b779328000bbfff4445498f86d15e468"} Oct 31 03:45:18.052194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.052172 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" Oct 31 03:45:18.052330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.052209 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr" event={"ID":"47656ece-d061-494c-9676-a31beceda2f5","Type":"ContainerDied","Data":"dcc345c88be3ddc7ed07b2f71c97369c40f152df553ece3a22051e1b29f9a16e"} Oct 31 03:45:18.052330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.052273 4488 scope.go:122] "RemoveContainer" containerID="254cd39f7e969c0ec7af2dcb471e64fa705260481d02f7aebddcdefcefb77270" Oct 31 03:45:18.348328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.347673 4488 scope.go:122] "RemoveContainer" containerID="8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047" Oct 31 03:45:18.395733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.395702 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:18.451849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.451815 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr"] Oct 31 03:45:18.457587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.457556 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-096812-29890305-kzxtr"] Oct 31 03:45:18.468780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.468747 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4\" (UniqueName: \"kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4\") pod \"b3186ddc-193c-42c7-9122-b61afd9f7944\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " Oct 31 03:45:18.468931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.468802 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca\") pod \"b3186ddc-193c-42c7-9122-b61afd9f7944\" (UID: \"b3186ddc-193c-42c7-9122-b61afd9f7944\") " Oct 31 03:45:18.470083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.470055 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca" pod "b3186ddc-193c-42c7-9122-b61afd9f7944" (UID: "b3186ddc-193c-42c7-9122-b61afd9f7944"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:18.472225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.472201 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4" pod "b3186ddc-193c-42c7-9122-b61afd9f7944" (UID: "b3186ddc-193c-42c7-9122-b61afd9f7944"). InnerVolumeSpecName "kube-api-access-4dwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:18.559045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.558485 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:18.571045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.571019 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwr4\" (UniqueName: \"kubernetes.io/projected/b3186ddc-193c-42c7-9122-b61afd9f7944-kube-api-access-4dwr4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:18.571045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.571047 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3186ddc-193c-42c7-9122-b61afd9f7944-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:18.672269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.672237 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf\" (UniqueName: \"kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf\") pod \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " Oct 31 03:45:18.672447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.672277 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca\") pod \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\" (UID: \"6f76e8f2-d978-41d3-85d7-df75b8611aeb\") " Oct 31 03:45:18.672682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.672655 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca" pod "6f76e8f2-d978-41d3-85d7-df75b8611aeb" (UID: "6f76e8f2-d978-41d3-85d7-df75b8611aeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:18.672858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.672844 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f76e8f2-d978-41d3-85d7-df75b8611aeb-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:18.724113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.722328 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf" pod "6f76e8f2-d978-41d3-85d7-df75b8611aeb" (UID: "6f76e8f2-d978-41d3-85d7-df75b8611aeb"). InnerVolumeSpecName "kube-api-access-bfzzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:18.774764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.774729 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzzf\" (UniqueName: \"kubernetes.io/projected/6f76e8f2-d978-41d3-85d7-df75b8611aeb-kube-api-access-bfzzf\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:18.793983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.793954 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:18.794169 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.794009 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:18.872357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:18.872319 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:19.054502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.054467 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz"] Oct 31 03:45:19.060270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.060237 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5429b4a8-zhqzz"] Oct 31 03:45:19.065366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.065338 4488 generic.go:356] "Generic (PLEG): container finished" podID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerID="1466f008c3fad064a6ac2789e6e59061b779328000bbfff4445498f86d15e468" exitCode=60 Oct 31 03:45:19.065504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.065412 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerDied","Data":"1466f008c3fad064a6ac2789e6e59061b779328000bbfff4445498f86d15e468"} Oct 31 03:45:19.065504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.065484 4488 scope.go:122] "RemoveContainer" containerID="9f641a9df38b67837e5afa5f5903387b03e8393758a2435accdeba7068ed760f" Oct 31 03:45:19.066040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.065933 4488 scope.go:122] "RemoveContainer" containerID="1466f008c3fad064a6ac2789e6e59061b779328000bbfff4445498f86d15e468" Oct 31 03:45:19.066400 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:19.066377 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4_odh-ai-gateway-infra(03f31640-b60c-4ffa-9a19-f8ad3d84638a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" Oct 31 03:45:19.067524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.067496 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" Oct 31 03:45:19.067620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.067551 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w" event={"ID":"b3186ddc-193c-42c7-9122-b61afd9f7944","Type":"ContainerDied","Data":"d62ce00e4f3249cacb28be9758ef1292b3056a0af6227a7a719144a13bbac137"} Oct 31 03:45:19.069958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.069926 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerStarted","Data":"cb0a98b87a2b9fcd87e49fe4e50551b5a5b6bfe57cd7095def483923fcb84a7c"} Oct 31 03:45:19.077407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.076743 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" Oct 31 03:45:19.077407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.076786 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv" event={"ID":"6f76e8f2-d978-41d3-85d7-df75b8611aeb","Type":"ContainerDied","Data":"bf8df7c0c9597ad7d17f00a0ad999edf30d3f94e4dd025ba34c836c0c74f78f6"} Oct 31 03:45:19.082394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.082371 4488 scope.go:122] "RemoveContainer" containerID="eb6707b99fc70973186d7b3267cfa4f97bbfda3e9704625915ef10bf0198fad9" Oct 31 03:45:19.093338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.093319 4488 scope.go:122] "RemoveContainer" containerID="e17b500cae316336a5151a24ca00a5045671ac887a26ca3ddd5e12207eadd1fd" Oct 31 03:45:19.131702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.131611 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:19.177196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.177162 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4"] Oct 31 03:45:19.333243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.333147 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:19.437284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.437168 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5"] Oct 31 03:45:19.438007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.437984 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:19.438007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438006 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438022 4488 cpu_manager.go:414] "containerMap: removing container" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438028 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438045 4488 cpu_manager.go:414] "containerMap: removing container" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438049 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438055 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438059 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438067 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438072 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438080 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438101 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438111 4488 cpu_manager.go:414] "containerMap: removing container" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438115 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438122 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438125 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438302 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438319 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438326 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438337 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438346 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438353 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="47656ece-d061-494c-9676-a31beceda2f5" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438361 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" containerName="cleanup" Oct 31 03:45:19.438712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.438372 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:19.444665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.444641 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.450993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.450966 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv"] Oct 31 03:45:19.464042 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.463990 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b63425-29890305-j9qfv"] Oct 31 03:45:19.487112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.486905 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.486951 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487140 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487219 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487331 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487399 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487515 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hf4\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487565 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.487679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.487599 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.495064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.495036 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5"] Oct 31 03:45:19.497436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.497416 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71186061-openshift-default-dockercfg-g8st8\"" Oct 31 03:45:19.533781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.533747 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:19.589287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.589243 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.589317 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.589417 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.589470 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590062 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.589788 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590197 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hf4\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590251 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590278 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590350 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590366 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590462 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.590711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590691 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.591019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.590997 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.592370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.592348 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.592658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.592637 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.595297 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.595277 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w"] Oct 31 03:45:19.601687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.601664 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890305-mkc7w"] Oct 31 03:45:19.652817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.652787 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.652987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.652827 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hf4\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4\") pod \"e2e-mt-71186061-openshift-default-5698544949-klmg5\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.706069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.705932 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:19.766480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.765999 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:19.795506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.795067 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:19.795506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:19.795147 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:20.110139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.109309 4488 generic.go:356] "Generic (PLEG): container finished" podID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerID="cb0a98b87a2b9fcd87e49fe4e50551b5a5b6bfe57cd7095def483923fcb84a7c" exitCode=60 Oct 31 03:45:20.110139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.109529 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerDied","Data":"cb0a98b87a2b9fcd87e49fe4e50551b5a5b6bfe57cd7095def483923fcb84a7c"} Oct 31 03:45:20.110139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.109587 4488 scope.go:122] "RemoveContainer" containerID="8b47436273ce7ede5a076e288b6639580cf27bbaf672ed5cf9be067818820047" Oct 31 03:45:20.110139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.110060 4488 scope.go:122] "RemoveContainer" containerID="cb0a98b87a2b9fcd87e49fe4e50551b5a5b6bfe57cd7095def483923fcb84a7c" Oct 31 03:45:20.110806 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:20.110527 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d_odh-ai-gateway-infra(361dffd6-4017-4026-b972-c04f9c49ce1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" Oct 31 03:45:20.287377 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:20.287319 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0761e73a_f63e_4edc_958a_b96db8614d28.slice/crio-1b5b76e37275901ab449eb28c60f5a5546a9a7e0139c32c2d623994a1692d72a WatchSource:0}: Error finding container 1b5b76e37275901ab449eb28c60f5a5546a9a7e0139c32c2d623994a1692d72a: Status 404 returned error can't find the container with id 1b5b76e37275901ab449eb28c60f5a5546a9a7e0139c32c2d623994a1692d72a Oct 31 03:45:20.293703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.293644 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:20.293843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.293752 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:20.293843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.293795 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:20.294403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.294299 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5"] Oct 31 03:45:20.369669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.358132 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca" path="/var/lib/kubelet/pods/2bdab6af-cac4-4bbb-9395-a1f2dee9a5ca/volumes" Oct 31 03:45:20.369669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.358942 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47656ece-d061-494c-9676-a31beceda2f5" path="/var/lib/kubelet/pods/47656ece-d061-494c-9676-a31beceda2f5/volumes" Oct 31 03:45:20.369669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.359698 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" path="/var/lib/kubelet/pods/6f76e8f2-d978-41d3-85d7-df75b8611aeb/volumes" Oct 31 03:45:20.369669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.360893 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3186ddc-193c-42c7-9122-b61afd9f7944" path="/var/lib/kubelet/pods/b3186ddc-193c-42c7-9122-b61afd9f7944/volumes" Oct 31 03:45:20.369669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.361388 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d"] Oct 31 03:45:20.465177 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.465137 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:20.535192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.512939 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6\" (UniqueName: \"kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6\") pod \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " Oct 31 03:45:20.535192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.513127 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca\") pod \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\" (UID: \"03f31640-b60c-4ffa-9a19-f8ad3d84638a\") " Oct 31 03:45:20.535672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.535637 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca" pod "03f31640-b60c-4ffa-9a19-f8ad3d84638a" (UID: "03f31640-b60c-4ffa-9a19-f8ad3d84638a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:20.544688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.544639 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6" pod "03f31640-b60c-4ffa-9a19-f8ad3d84638a" (UID: "03f31640-b60c-4ffa-9a19-f8ad3d84638a"). InnerVolumeSpecName "kube-api-access-jf9d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:20.640162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.635812 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9d6\" (UniqueName: \"kubernetes.io/projected/03f31640-b60c-4ffa-9a19-f8ad3d84638a-kube-api-access-jf9d6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:20.640162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.635851 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f31640-b60c-4ffa-9a19-f8ad3d84638a-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:20.793955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.793919 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:20.794131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:20.793983 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:21.144563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.144531 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" Oct 31 03:45:21.144563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.144541 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4" event={"ID":"03f31640-b60c-4ffa-9a19-f8ad3d84638a","Type":"ContainerDied","Data":"baf46551aa2cf5957bbf1b62c9711db54b596ffba61d243a30b1a750001ceb06"} Oct 31 03:45:21.145150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.144630 4488 scope.go:122] "RemoveContainer" containerID="1466f008c3fad064a6ac2789e6e59061b779328000bbfff4445498f86d15e468" Oct 31 03:45:21.152952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.152918 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" event={"ID":"0761e73a-f63e-4edc-958a-b96db8614d28","Type":"ContainerStarted","Data":"1528241113c33e48afc9d321875167590fd98ed89d2437a9dad6c4a597a4a8a1"} Oct 31 03:45:21.153073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.152996 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" event={"ID":"0761e73a-f63e-4edc-958a-b96db8614d28","Type":"ContainerStarted","Data":"1b5b76e37275901ab449eb28c60f5a5546a9a7e0139c32c2d623994a1692d72a"} Oct 31 03:45:21.385380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.385351 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:21.430656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.430561 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:21.448664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.448629 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca\") pod \"361dffd6-4017-4026-b972-c04f9c49ce1b\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " Oct 31 03:45:21.448828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.448748 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd\" (UniqueName: \"kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd\") pod \"361dffd6-4017-4026-b972-c04f9c49ce1b\" (UID: \"361dffd6-4017-4026-b972-c04f9c49ce1b\") " Oct 31 03:45:21.449061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.449038 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca" pod "361dffd6-4017-4026-b972-c04f9c49ce1b" (UID: "361dffd6-4017-4026-b972-c04f9c49ce1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:21.449679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.449655 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361dffd6-4017-4026-b972-c04f9c49ce1b-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:21.449858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.449799 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podStartSLOduration=2.449784742 podStartE2EDuration="2.449784742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:21.335524145 +0000 UTC m=+1549.132242414" watchObservedRunningTime="2026-10-31 03:45:21.449784742 +0000 UTC m=+1549.246502999" Oct 31 03:45:21.451490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.451461 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd" pod "361dffd6-4017-4026-b972-c04f9c49ce1b" (UID: "361dffd6-4017-4026-b972-c04f9c49ce1b"). InnerVolumeSpecName "kube-api-access-gssxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:21.452511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.452490 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4"] Oct 31 03:45:21.457451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.457421 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b98893-29890305-m4cg4"] Oct 31 03:45:21.551490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.551453 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gssxd\" (UniqueName: \"kubernetes.io/projected/361dffd6-4017-4026-b972-c04f9c49ce1b-kube-api-access-gssxd\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:45:21.592747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.592718 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:21.771178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.770392 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:21.778713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.778685 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:21.793793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.793765 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:21.794024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:21.794002 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:22.165458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.165431 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" Oct 31 03:45:22.165937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.165456 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d" event={"ID":"361dffd6-4017-4026-b972-c04f9c49ce1b","Type":"ContainerDied","Data":"2227153f8292f15aa40967d203ca31d419b4dc9acbe31b7775e056428e3b37c2"} Oct 31 03:45:22.165937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.165540 4488 scope.go:122] "RemoveContainer" containerID="cb0a98b87a2b9fcd87e49fe4e50551b5a5b6bfe57cd7095def483923fcb84a7c" Oct 31 03:45:22.166369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.166343 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:22.171369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.171344 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:45:22.354211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.350052 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" path="/var/lib/kubelet/pods/03f31640-b60c-4ffa-9a19-f8ad3d84638a/volumes" Oct 31 03:45:22.354211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.350550 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d"] Oct 31 03:45:22.355172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.355143 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890305-wpr4d"] Oct 31 03:45:22.729114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.728878 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:22.793889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.793858 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:22.794046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.793911 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:22.881872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:22.881838 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:23.054359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:23.054268 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt"] Oct 31 03:45:23.059260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:23.059231 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaa3bc7-gc6bt"] Oct 31 03:45:23.793283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:23.793242 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 03:45:23.793803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:23.793301 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 03:45:24.052313 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:24.052203 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj"] Oct 31 03:45:24.107020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:24.106974 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81898ccb-2lxcj"] Oct 31 03:45:24.319682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:24.319582 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" path="/var/lib/kubelet/pods/361dffd6-4017-4026-b972-c04f9c49ce1b/volumes" Oct 31 03:45:24.320232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:24.320213 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5605ed00-fdcc-4d12-8355-1fbc8e4ba903" path="/var/lib/kubelet/pods/5605ed00-fdcc-4d12-8355-1fbc8e4ba903/volumes" Oct 31 03:45:24.320696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:24.320680 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5" path="/var/lib/kubelet/pods/bb0942e8-57e0-49b4-8d8f-1dfe6354f8d5/volumes" Oct 31 03:45:24.924888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:24.924856 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:45:25.408644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:25.408552 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:25.571525 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:25.571490 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:25.572248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:25.572222 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:25.573150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:25.573131 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" Oct 31 03:45:25.592639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:25.592609 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:25.789218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:25.788536 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:26.896051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:26.896013 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:27.052368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:27.052332 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:27.236174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:27.235970 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:29.484758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:29.484716 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:29.651876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:29.651843 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:30.061024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.060969 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc"] Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.258836 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d9b98e-mgfxc"] Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.260373 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz"] Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261381 4488 cpu_manager.go:414] "containerMap: removing container" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261398 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261419 4488 cpu_manager.go:414] "containerMap: removing container" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261425 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261441 4488 cpu_manager.go:414] "containerMap: removing container" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261448 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261461 4488 cpu_manager.go:414] "containerMap: removing container" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261468 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261480 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261486 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261495 4488 cpu_manager.go:414] "containerMap: removing container" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261501 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261655 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261666 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261673 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261684 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:45:30.263119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.261693 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f76e8f2-d978-41d3-85d7-df75b8611aeb" containerName="cleanup" Oct 31 03:45:30.268530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.268503 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.296987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.296952 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz"] Oct 31 03:45:30.329437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.329409 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fdcf82-327a-4770-82e1-2e56d89ddcc7" path="/var/lib/kubelet/pods/11fdcf82-327a-4770-82e1-2e56d89ddcc7/volumes" Oct 31 03:45:30.361989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.361955 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362120 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362175 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362254 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362211 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362360 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362408 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362548 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362643 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.362691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.362665 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnc9\" (UniqueName: \"kubernetes.io/projected/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-kube-api-access-xgnc9\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464008 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464151 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464211 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464363 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464326 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464361 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnc9\" (UniqueName: \"kubernetes.io/projected/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-kube-api-access-xgnc9\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464487 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464706 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464679 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464728 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.464814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464741 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.465176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464836 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.465176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.464880 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.465176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.465067 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.465325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.465288 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.465426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.465402 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.466953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.466926 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.467200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.467181 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.525824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.525789 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.526207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.525872 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnc9\" (UniqueName: \"kubernetes.io/projected/7f7842f2-5eeb-417d-8bcb-76f99f23fe23-kube-api-access-xgnc9\") pod \"maas-default-gateway-openshift-default-fd4b99879-znzrz\" (UID: \"7f7842f2-5eeb-417d-8bcb-76f99f23fe23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:30.597530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:30.597450 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:31.066626 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:45:31.066595 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7842f2_5eeb_417d_8bcb_76f99f23fe23.slice/crio-f743a822900271dc332b1de29abf83f577a53cec69058cacca0e65bc1f596374 WatchSource:0}: Error finding container f743a822900271dc332b1de29abf83f577a53cec69058cacca0e65bc1f596374: Status 404 returned error can't find the container with id f743a822900271dc332b1de29abf83f577a53cec69058cacca0e65bc1f596374 Oct 31 03:45:31.067033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.067009 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz"] Oct 31 03:45:31.070159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.070122 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:31.070264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.070217 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:31.070335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.070271 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:45:31.343240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.343134 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" event={"ID":"7f7842f2-5eeb-417d-8bcb-76f99f23fe23","Type":"ContainerStarted","Data":"d99880eb43adfbe0d9d32ff9991d9181b29bd7eb2fdeaa31cfb0c9028f19d459"} Oct 31 03:45:31.343240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.343201 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" event={"ID":"7f7842f2-5eeb-417d-8bcb-76f99f23fe23","Type":"ContainerStarted","Data":"f743a822900271dc332b1de29abf83f577a53cec69058cacca0e65bc1f596374"} Oct 31 03:45:31.486113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.481374 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podStartSLOduration=1.48135244 podStartE2EDuration="1.48135244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:31.464575134 +0000 UTC m=+1559.261293399" watchObservedRunningTime="2026-10-31 03:45:31.48135244 +0000 UTC m=+1559.278070699" Oct 31 03:45:31.598427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:31.598332 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:32.203393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:32.203337 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:32.599976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:32.599920 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:32.600431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:32.600008 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:33.598103 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:33.598061 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:33.598297 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:33.598139 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:33.991280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:33.991202 4488 scope.go:122] "RemoveContainer" containerID="2e91d71ebd87979c9492a83a261f31ba5372b4257581d5bf716f5407ef99e0b0" Oct 31 03:45:34.019432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:34.019267 4488 scope.go:122] "RemoveContainer" containerID="92f2a2355a94231c09d1164a2e7b616a73b489948b6f55f55088e22354fe5d16" Oct 31 03:45:34.598505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:34.598456 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:34.598704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:34.598541 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:35.363833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:35.363801 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:35.598452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:35.598408 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:35.598656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:35.598497 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:36.050126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:36.050059 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz"] Oct 31 03:45:36.099966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:36.099903 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ab031f-nlccz"] Oct 31 03:45:36.338522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:36.338490 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bbd886-2b3c-4a4c-8592-b6b8665a2e48" path="/var/lib/kubelet/pods/a0bbd886-2b3c-4a4c-8592-b6b8665a2e48/volumes" Oct 31 03:45:36.590592 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:45:36.590505 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:45:36.598109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:36.598063 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:36.598326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:36.598134 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:37.598506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:37.598464 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:37.599007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:37.598533 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:37.678913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:37.678861 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:38.597912 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:38.597860 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 03:45:38.598190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:38.597929 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 03:45:39.683427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:39.683390 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:39.683966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:39.683933 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:39.684892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:39.684870 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" Oct 31 03:45:42.712081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:42.712023 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:43.052981 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:43.052895 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz"] Oct 31 03:45:43.105316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:43.103623 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm"] Oct 31 03:45:43.109008 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:43.108978 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c18149-6smrz"] Oct 31 03:45:43.113680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:43.113658 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d"] Oct 31 03:45:43.118248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:43.118215 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb70e33-tzplm"] Oct 31 03:45:43.153038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:43.153007 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d325dc96-72q5d"] Oct 31 03:45:44.320194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:44.320106 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f49d4c-3032-4885-b74d-d6326c92fae3" path="/var/lib/kubelet/pods/11f49d4c-3032-4885-b74d-d6326c92fae3/volumes" Oct 31 03:45:44.320652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:44.320638 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133c1810-55c4-484b-9d1d-da01cc0084bb" path="/var/lib/kubelet/pods/133c1810-55c4-484b-9d1d-da01cc0084bb/volumes" Oct 31 03:45:44.321102 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:44.321075 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c3f234-ecdc-4739-a6e7-de4ee34c65fd" path="/var/lib/kubelet/pods/d5c3f234-ecdc-4739-a6e7-de4ee34c65fd/volumes" Oct 31 03:45:45.049813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:45.049770 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c"] Oct 31 03:45:45.099325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:45.099287 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629191e4-nxl6c"] Oct 31 03:45:46.318099 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:46.318057 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf86bcd-a193-4af1-b42d-8efa7c23789d" path="/var/lib/kubelet/pods/4bf86bcd-a193-4af1-b42d-8efa7c23789d/volumes" Oct 31 03:45:47.619495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:47.619428 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:47.850837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:47.850800 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:48.007424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:48.007339 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:48.051834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:48.051789 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq"] Oct 31 03:45:48.100132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:48.100074 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b50722-4rzbq"] Oct 31 03:45:48.319674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:48.319593 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe68c202-5b09-4072-b6ad-8437b992cb08" path="/var/lib/kubelet/pods/fe68c202-5b09-4072-b6ad-8437b992cb08/volumes" Oct 31 03:45:49.050932 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:49.050895 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l"] Oct 31 03:45:49.101258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:49.101222 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d50d52-8gg4l"] Oct 31 03:45:50.319323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:50.319290 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652a02e6-d553-4394-9e13-46a916638f52" path="/var/lib/kubelet/pods/652a02e6-d553-4394-9e13-46a916638f52/volumes" Oct 31 03:45:52.412040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:52.411928 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:53.053932 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:53.053892 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m"] Oct 31 03:45:53.103508 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:53.103473 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc73754-2hn4m"] Oct 31 03:45:54.336877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:54.336842 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692e81fc-bdd4-4a7d-82c4-093f3d61a84b" path="/var/lib/kubelet/pods/692e81fc-bdd4-4a7d-82c4-093f3d61a84b/volumes" Oct 31 03:45:57.212115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:57.204727 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:57.981584 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:57.981543 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m"] Oct 31 03:45:57.981960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:57.981917 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" containerID="cri-o://a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963" gracePeriod=30 Oct 31 03:45:57.999237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:57.999204 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6"] Oct 31 03:45:57.999487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:57.999461 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" containerID="cri-o://bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9" gracePeriod=30 Oct 31 03:45:58.029347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:58.029305 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj"] Oct 31 03:45:58.029885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:58.029827 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" containerID="cri-o://018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea" gracePeriod=30 Oct 31 03:45:58.036349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:58.036319 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98"] Oct 31 03:45:58.036593 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:58.036569 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" containerID="cri-o://ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902" gracePeriod=30 Oct 31 03:45:59.287291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:59.287254 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:45:59.287757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:45:59.287332 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:01.050707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:01.050670 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm"] Oct 31 03:46:01.100642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:01.100610 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0021d397-mjkzm"] Oct 31 03:46:01.535648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:01.535613 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:46:01.535826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:01.535675 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:46:02.004921 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:02.004886 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:02.272809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:02.272721 4488 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:46:02.272809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:02.272784 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:46:02.321740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:02.321706 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873d976a-5727-4493-87d1-17c3c92933ac" path="/var/lib/kubelet/pods/873d976a-5727-4493-87d1-17c3c92933ac/volumes" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.474586 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.531434 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.582434 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.588807 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646416 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646542 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646592 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646624 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646650 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646693 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646725 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646779 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646803 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646838 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646866 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646895 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646928 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.646976 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647008 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647010 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647010 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647049 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647221 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647249 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647276 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647309 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647342 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647365 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647382 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647442 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647469 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647476 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647482 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647513 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647545 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647597 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647610 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647624 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647685 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647730 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647776 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647817 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647816 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.647975 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs\") pod \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\" (UID: \"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648003 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648042 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648162 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs\") pod \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\" (UID: \"2f2ae28f-266b-4f09-a723-d4baa8e887cf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648201 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj\") pod \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\" (UID: \"a7c60e57-0008-4c21-8157-ebaa654a3fbf\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648281 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs\") pod \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\" (UID: \"daf2fbea-df19-478b-b3b4-d4e86fa891c6\") " Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648322 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648440 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648511 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648693 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648785 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.648994 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649083 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649199 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649304 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649322 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649331 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649343 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649353 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649361 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649362 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649370 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649382 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649396 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649406 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649417 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649425 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649433 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649442 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649450 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649461 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649647 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.649693 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.656574 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.656585 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.658954 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "kube-api-access-wc7p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.659243 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "kube-api-access-ktvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.659589 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.659992 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.660428 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.660675 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.660923 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.661164 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token" pod "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" (UID: "4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.661357 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "kube-api-access-tbqwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.661475 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.661499 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.661806 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token" pod "2f2ae28f-266b-4f09-a723-d4baa8e887cf" (UID: "2f2ae28f-266b-4f09-a723-d4baa8e887cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.661868 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj" pod "a7c60e57-0008-4c21-8157-ebaa654a3fbf" (UID: "a7c60e57-0008-4c21-8157-ebaa654a3fbf"). InnerVolumeSpecName "kube-api-access-5v6vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.686111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.662205 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token" pod "daf2fbea-df19-478b-b3b4-d4e86fa891c6" (UID: "daf2fbea-df19-478b-b3b4-d4e86fa891c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750634 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750664 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqwq\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-kube-api-access-tbqwq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750676 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750685 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750694 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750703 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvmq\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-kube-api-access-ktvmq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750711 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc7p5\" (UniqueName: \"kubernetes.io/projected/2f2ae28f-266b-4f09-a723-d4baa8e887cf-kube-api-access-wc7p5\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.750713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750720 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750729 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750736 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c60e57-0008-4c21-8157-ebaa654a3fbf-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750746 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750755 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750763 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750771 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750779 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v6vj\" (UniqueName: \"kubernetes.io/projected/a7c60e57-0008-4c21-8157-ebaa654a3fbf-kube-api-access-5v6vj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750787 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf2fbea-df19-478b-b3b4-d4e86fa891c6-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750795 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750802 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf2fbea-df19-478b-b3b4-d4e86fa891c6-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750810 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.751078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.750819 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f2ae28f-266b-4f09-a723-d4baa8e887cf-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.803931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.803891 4488 generic.go:356] "Generic (PLEG): container finished" podID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerID="a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963" exitCode=0 Oct 31 03:46:03.804209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.803988 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" Oct 31 03:46:03.804209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.804037 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" event={"ID":"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21","Type":"ContainerDied","Data":"a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963"} Oct 31 03:46:03.804209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.804134 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m" event={"ID":"4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21","Type":"ContainerDied","Data":"18f40c2727e11767f5c64b4929ffb2fd8aa35643ff3187d48f344b7db4c33f70"} Oct 31 03:46:03.804209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.804172 4488 scope.go:122] "RemoveContainer" containerID="a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963" Oct 31 03:46:03.807524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.807497 4488 generic.go:356] "Generic (PLEG): container finished" podID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerID="bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9" exitCode=0 Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.807715 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" event={"ID":"2f2ae28f-266b-4f09-a723-d4baa8e887cf","Type":"ContainerDied","Data":"bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9"} Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.807757 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" event={"ID":"2f2ae28f-266b-4f09-a723-d4baa8e887cf","Type":"ContainerDied","Data":"a58b91f6f250bbe690aeccdef5acb80d32407d4511501424f32f3b83a642324e"} Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.807774 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6" Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.810959 4488 generic.go:356] "Generic (PLEG): container finished" podID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerID="ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902" exitCode=0 Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.811044 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.811106 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" event={"ID":"daf2fbea-df19-478b-b3b4-d4e86fa891c6","Type":"ContainerDied","Data":"ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902"} Oct 31 03:46:03.815314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.811149 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98" event={"ID":"daf2fbea-df19-478b-b3b4-d4e86fa891c6","Type":"ContainerDied","Data":"f714de5d18f6d4c30539da1b119dc8e68beefa5e14252abf649b5bca9ae936af"} Oct 31 03:46:03.820457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.820429 4488 generic.go:356] "Generic (PLEG): container finished" podID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerID="018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea" exitCode=0 Oct 31 03:46:03.820608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.820473 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" Oct 31 03:46:03.820608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.820474 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" event={"ID":"a7c60e57-0008-4c21-8157-ebaa654a3fbf","Type":"ContainerDied","Data":"018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea"} Oct 31 03:46:03.820608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.820584 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj" event={"ID":"a7c60e57-0008-4c21-8157-ebaa654a3fbf","Type":"ContainerDied","Data":"8438d4590ce6376e2fb46fef4102bc0e8e2023f7b5c8dac60e723ae8f329e66c"} Oct 31 03:46:03.830847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.830827 4488 scope.go:122] "RemoveContainer" containerID="a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963" Oct 31 03:46:03.831849 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:03.831459 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963\": container with ID starting with a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963 not found: ID does not exist" containerID="a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963" Oct 31 03:46:03.831849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.831501 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963"} err="failed to get container status \"a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963\": rpc error: code = NotFound desc = could not find container \"a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963\": container with ID starting with a8f1d50d55dfb6695c6ad4caf52ab7f1221a91b0b3ed4e380ddc70620c6c2963 not found: ID does not exist" Oct 31 03:46:03.831849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.831524 4488 scope.go:122] "RemoveContainer" containerID="bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9" Oct 31 03:46:03.854883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.854854 4488 scope.go:122] "RemoveContainer" containerID="bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9" Oct 31 03:46:03.855759 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:03.855619 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9\": container with ID starting with bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9 not found: ID does not exist" containerID="bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9" Oct 31 03:46:03.855759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.855658 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9"} err="failed to get container status \"bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9\": rpc error: code = NotFound desc = could not find container \"bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9\": container with ID starting with bc504b8f26e93b6aaf209801cfac0cf1435085d5a458295baeb8b1f5e8717ab9 not found: ID does not exist" Oct 31 03:46:03.855759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.855677 4488 scope.go:122] "RemoveContainer" containerID="ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902" Oct 31 03:46:03.867889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.867868 4488 scope.go:122] "RemoveContainer" containerID="5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72" Oct 31 03:46:03.878396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.878375 4488 scope.go:122] "RemoveContainer" containerID="ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902" Oct 31 03:46:03.878747 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:03.878726 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902\": container with ID starting with ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902 not found: ID does not exist" containerID="ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902" Oct 31 03:46:03.878834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.878754 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902"} err="failed to get container status \"ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902\": rpc error: code = NotFound desc = could not find container \"ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902\": container with ID starting with ac669def6a1fd00abf9b4679614a922590014971c5de1dcc6685366cfd231902 not found: ID does not exist" Oct 31 03:46:03.878834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.878774 4488 scope.go:122] "RemoveContainer" containerID="5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72" Oct 31 03:46:03.879040 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:03.879027 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72\": container with ID starting with 5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72 not found: ID does not exist" containerID="5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72" Oct 31 03:46:03.879150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.879043 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72"} err="failed to get container status \"5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72\": rpc error: code = NotFound desc = could not find container \"5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72\": container with ID starting with 5fcb2c8054e5bb46fab529892a2055540ce3847e21808495c07f23fa124e6c72 not found: ID does not exist" Oct 31 03:46:03.879150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.879054 4488 scope.go:122] "RemoveContainer" containerID="018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea" Oct 31 03:46:03.903015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.902991 4488 scope.go:122] "RemoveContainer" containerID="018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea" Oct 31 03:46:03.904278 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:03.904248 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea\": container with ID starting with 018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea not found: ID does not exist" containerID="018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea" Oct 31 03:46:03.904412 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:03.904288 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea"} err="failed to get container status \"018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea\": rpc error: code = NotFound desc = could not find container \"018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea\": container with ID starting with 018955f649a1aa0668e88d1880644ec5fe18ebcadacd6e1a03293ecec1c581ea not found: ID does not exist" Oct 31 03:46:04.145365 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.145326 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98"] Oct 31 03:46:04.149633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.149610 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-vsg98"] Oct 31 03:46:04.252632 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.252550 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m"] Oct 31 03:46:04.259007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.258974 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-t848m"] Oct 31 03:46:04.319654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.319623 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" path="/var/lib/kubelet/pods/4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21/volumes" Oct 31 03:46:04.320219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.320203 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" path="/var/lib/kubelet/pods/daf2fbea-df19-478b-b3b4-d4e86fa891c6/volumes" Oct 31 03:46:04.369388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.369353 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6"] Oct 31 03:46:04.373488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.373467 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-9m2h6"] Oct 31 03:46:04.483409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.483374 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj"] Oct 31 03:46:04.487485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.487460 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-z77mj"] Oct 31 03:46:04.833651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.833620 4488 generic.go:356] "Generic (PLEG): container finished" podID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerID="3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf" exitCode=0 Oct 31 03:46:04.834184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.833709 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" event={"ID":"78b4571b-f72d-4116-afb5-f8c63fc23db7","Type":"ContainerDied","Data":"3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf"} Oct 31 03:46:04.834345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.834324 4488 scope.go:122] "RemoveContainer" containerID="3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf" Oct 31 03:46:04.836641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.836604 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:46:04.836770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.836720 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:46:04.836812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:04.836768 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:46:05.850726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:05.850693 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" event={"ID":"78b4571b-f72d-4116-afb5-f8c63fc23db7","Type":"ContainerStarted","Data":"8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428"} Oct 31 03:46:06.091140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:06.091068 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:46:06.091361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:06.091312 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:46:06.091442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:06.091367 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 03:46:06.338419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:06.338384 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" path="/var/lib/kubelet/pods/2f2ae28f-266b-4f09-a723-d4baa8e887cf/volumes" Oct 31 03:46:06.338939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:06.338924 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" path="/var/lib/kubelet/pods/a7c60e57-0008-4c21-8157-ebaa654a3fbf/volumes" Oct 31 03:46:06.591118 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:06.591007 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:46:06.829310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:06.824688 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:07.091402 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:07.091366 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:46:07.091799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:07.091427 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 03:46:08.092039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:08.091999 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:46:08.092580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:08.092073 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 03:46:09.050805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:09.050768 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6"] Oct 31 03:46:09.091412 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:09.091378 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:46:09.091598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:09.091432 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 03:46:09.100345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:09.100313 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77a243d-ph6t6"] Oct 31 03:46:10.091925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:10.091885 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:46:10.092151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:10.091964 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 03:46:10.337727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:10.337683 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394815d9-aa67-4f97-a741-e875f54fbbd6" path="/var/lib/kubelet/pods/394815d9-aa67-4f97-a741-e875f54fbbd6/volumes" Oct 31 03:46:11.091617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:11.091583 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:46:11.091815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:11.091646 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 03:46:11.690642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:11.690598 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:12.096874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:12.096842 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:46:12.097303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:12.097284 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:46:12.098144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:12.098083 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 03:46:12.442702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:12.442615 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:46:16.050652 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:16.050620 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd"] Oct 31 03:46:16.099866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:16.099831 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8702ed28-z6fnd"] Oct 31 03:46:16.162337 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:16.162291 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:16.320496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:16.320411 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562" path="/var/lib/kubelet/pods/3eafd08a-d3d7-4e8a-acd2-6ba95c0f7562/volumes" Oct 31 03:46:16.327110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:16.327056 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:16.484855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:16.484827 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:18.048794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:18.048759 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh"] Oct 31 03:46:18.098694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:18.098657 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837b6d4c-tlbnh"] Oct 31 03:46:18.320414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:18.320322 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e68a3ba-5351-4125-b422-c4f0dc042288" path="/var/lib/kubelet/pods/4e68a3ba-5351-4125-b422-c4f0dc042288/volumes" Oct 31 03:46:21.172856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.172808 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl"] Oct 31 03:46:21.173717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173694 4488 cpu_manager.go:414] "containerMap: removing container" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173731 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173747 4488 cpu_manager.go:414] "containerMap: removing container" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173754 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173788 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173795 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173806 4488 cpu_manager.go:414] "containerMap: removing container" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:46:21.173833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173813 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173857 4488 cpu_manager.go:414] "containerMap: removing container" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173864 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173880 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.173887 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174064 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f31640-b60c-4ffa-9a19-f8ad3d84638a" containerName="cleanup" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174082 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="361dffd6-4017-4026-b972-c04f9c49ce1b" containerName="cleanup" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174111 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2ae28f-266b-4f09-a723-d4baa8e887cf" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174122 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee9ef6f-23a4-40bc-af93-4d4a41a8bc21" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174135 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174143 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf2fbea-df19-478b-b3b4-d4e86fa891c6" containerName="istio-proxy" Oct 31 03:46:21.174240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.174155 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c60e57-0008-4c21-8157-ebaa654a3fbf" containerName="istio-proxy" Oct 31 03:46:21.179290 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.179268 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.235312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.235282 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl"] Oct 31 03:46:21.242767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.242716 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:46:21.242767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.242752 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:46:21.280212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.280181 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:46:21.280212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.280196 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:46:21.299575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.299545 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhnw\" (UniqueName: \"kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.299754 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.299582 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.299841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.299823 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.400995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.400957 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.401536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.401507 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.401761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.401737 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.401865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.401745 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhnw\" (UniqueName: \"kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.403917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.403891 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.463014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.462935 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhnw\" (UniqueName: \"kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw\") pod \"maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.493942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.493911 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:21.711046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.711014 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:21.901287 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:46:21.900079 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5606f8f4_b409_401d_b30d_43726a2a109a.slice/crio-9899597c82e58a2f445661e3929c6917fb25589aca088ef3e3668ef63730613f WatchSource:0}: Error finding container 9899597c82e58a2f445661e3929c6917fb25589aca088ef3e3668ef63730613f: Status 404 returned error can't find the container with id 9899597c82e58a2f445661e3929c6917fb25589aca088ef3e3668ef63730613f Oct 31 03:46:21.913636 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:21.901553 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl"] Oct 31 03:46:22.040821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:22.040775 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" event={"ID":"5606f8f4-b409-401d-b30d-43726a2a109a","Type":"ContainerStarted","Data":"9899597c82e58a2f445661e3929c6917fb25589aca088ef3e3668ef63730613f"} Oct 31 03:46:23.052926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:23.052889 4488 generic.go:356] "Generic (PLEG): container finished" podID="5606f8f4-b409-401d-b30d-43726a2a109a" containerID="fb2d71ddf22c55ef37b6d3b30082fe9406121e8b1e7c2ea01023ad73b5c78ada" exitCode=0 Oct 31 03:46:23.053371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:23.052974 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" event={"ID":"5606f8f4-b409-401d-b30d-43726a2a109a","Type":"ContainerDied","Data":"fb2d71ddf22c55ef37b6d3b30082fe9406121e8b1e7c2ea01023ad73b5c78ada"} Oct 31 03:46:24.050081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.050041 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5"] Oct 31 03:46:24.100604 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.100571 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1ea513-lnzd5"] Oct 31 03:46:24.249182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.249157 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:24.319229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.319152 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf01b115-5196-42c5-950b-4dff7db06a0e" path="/var/lib/kubelet/pods/cf01b115-5196-42c5-950b-4dff7db06a0e/volumes" Oct 31 03:46:24.355005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.354973 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca\") pod \"5606f8f4-b409-401d-b30d-43726a2a109a\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " Oct 31 03:46:24.355248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.355068 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw\" (UniqueName: \"kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw\") pod \"5606f8f4-b409-401d-b30d-43726a2a109a\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " Oct 31 03:46:24.355248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.355126 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca\") pod \"5606f8f4-b409-401d-b30d-43726a2a109a\" (UID: \"5606f8f4-b409-401d-b30d-43726a2a109a\") " Oct 31 03:46:24.355401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.355370 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca" pod "5606f8f4-b409-401d-b30d-43726a2a109a" (UID: "5606f8f4-b409-401d-b30d-43726a2a109a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:24.356004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.355984 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.357503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.357481 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca" pod "5606f8f4-b409-401d-b30d-43726a2a109a" (UID: "5606f8f4-b409-401d-b30d-43726a2a109a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:24.357555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.357519 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw" pod "5606f8f4-b409-401d-b30d-43726a2a109a" (UID: "5606f8f4-b409-401d-b30d-43726a2a109a"). InnerVolumeSpecName "kube-api-access-vmhnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:24.458158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.458123 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmhnw\" (UniqueName: \"kubernetes.io/projected/5606f8f4-b409-401d-b30d-43726a2a109a-kube-api-access-vmhnw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.458158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:24.458159 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5606f8f4-b409-401d-b30d-43726a2a109a-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:25.073627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:25.073597 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" Oct 31 03:46:25.073843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:25.073631 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl" event={"ID":"5606f8f4-b409-401d-b30d-43726a2a109a","Type":"ContainerDied","Data":"9899597c82e58a2f445661e3929c6917fb25589aca088ef3e3668ef63730613f"} Oct 31 03:46:25.073843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:25.073685 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9899597c82e58a2f445661e3929c6917fb25589aca088ef3e3668ef63730613f" Oct 31 03:46:26.051205 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.051168 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:26.773725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.773633 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f"] Oct 31 03:46:26.774359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.774345 4488 cpu_manager.go:414] "containerMap: removing container" podUID="5606f8f4-b409-401d-b30d-43726a2a109a" containerName="revoke-keys" Oct 31 03:46:26.774410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.774361 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5606f8f4-b409-401d-b30d-43726a2a109a" containerName="revoke-keys" Oct 31 03:46:26.774512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.774502 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="5606f8f4-b409-401d-b30d-43726a2a109a" containerName="revoke-keys" Oct 31 03:46:26.777923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.777904 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:26.832355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.831968 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71186061-serving-cert\"" Oct 31 03:46:26.881867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.881830 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\"" Oct 31 03:46:26.881867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.881833 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:46:26.881867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.881833 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:46:26.882258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.881922 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:46:26.897816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.897772 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f"] Oct 31 03:46:26.908643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.908608 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:26.908798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.908719 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:26.908871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.908818 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:26.908911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:26.908902 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmjw\" (UniqueName: \"kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.011210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.011173 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.011413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.011250 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.011413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.011304 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.011413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.011375 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmjw\" (UniqueName: \"kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.011668 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:27.011455 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-71186061-serving-cert: secret "maas-api-e2e-mt-71186061-serving-cert" not found Oct 31 03:46:27.011668 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:27.011537 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls podName:190d507f-0545-429c-9ee6-9c7f7c081b66 nodeName:}" failed. No retries permitted until 2026-10-31 03:46:27.511513613 +0000 UTC m=+1615.308231869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls") pod "maas-api-e2e-mt-71186061-774c54847b-8kb7f" (UID: "190d507f-0545-429c-9ee6-9c7f7c081b66") : secret "maas-api-e2e-mt-71186061-serving-cert" not found Oct 31 03:46:27.012051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.012026 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.014392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.014367 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.034713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.034614 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:27.080729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.080661 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmjw\" (UniqueName: \"kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.193341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.193302 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:27.346790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.346752 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:27.521785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.521743 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.524456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.524430 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\") pod \"maas-api-e2e-mt-71186061-774c54847b-8kb7f\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:27.693588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:27.693494 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:28.089557 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:46:28.089520 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190d507f_0545_429c_9ee6_9c7f7c081b66.slice/crio-8d85cea182d0fd6e8d5b80d8c30803bebef9cb5b42c4edef42daf09f6d0e79e0 WatchSource:0}: Error finding container 8d85cea182d0fd6e8d5b80d8c30803bebef9cb5b42c4edef42daf09f6d0e79e0: Status 404 returned error can't find the container with id 8d85cea182d0fd6e8d5b80d8c30803bebef9cb5b42c4edef42daf09f6d0e79e0 Oct 31 03:46:28.089986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:28.089856 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f"] Oct 31 03:46:28.155644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:28.155607 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" event={"ID":"190d507f-0545-429c-9ee6-9c7f7c081b66","Type":"ContainerStarted","Data":"8d85cea182d0fd6e8d5b80d8c30803bebef9cb5b42c4edef42daf09f6d0e79e0"} Oct 31 03:46:31.051941 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:31.051896 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84"] Oct 31 03:46:31.056939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:31.056905 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7433c1c9-58f84"] Oct 31 03:46:32.204777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:32.204735 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" event={"ID":"190d507f-0545-429c-9ee6-9c7f7c081b66","Type":"ContainerStarted","Data":"e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b"} Oct 31 03:46:32.205323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:32.204843 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:32.320471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:32.320438 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a112b709-d618-4b06-9592-1b9983b56daa" path="/var/lib/kubelet/pods/a112b709-d618-4b06-9592-1b9983b56daa/volumes" Oct 31 03:46:32.323788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:32.323740 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" podStartSLOduration=3.228893106 podStartE2EDuration="6.323723807s" totalImagesPullingTime="3.094830701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:32.322795014 +0000 UTC m=+1620.119513275" watchObservedRunningTime="2026-10-31 03:46:32.323723807 +0000 UTC m=+1620.120442063" Oct 31 03:46:32.339285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:32.339252 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:32.682881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:32.682845 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:33.054249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:33.054160 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g"] Oct 31 03:46:33.057818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:33.057793 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3921a4a6-kwj8g"] Oct 31 03:46:34.141268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.141237 4488 scope.go:122] "RemoveContainer" containerID="57150e8f76dff86e36e115b976062bef567edca763b6ec045e92cb23d16037f9" Oct 31 03:46:34.152682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.152657 4488 scope.go:122] "RemoveContainer" containerID="9ca8c6f02d97999bf755196abfa961e87755e6ccd0141632ed78e91310d083f6" Oct 31 03:46:34.163874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.163854 4488 scope.go:122] "RemoveContainer" containerID="820cda6663b18efeb4136c3db812f5d8927b3c3340a74e11bb8e7f7c0979ce6d" Oct 31 03:46:34.174532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.174513 4488 scope.go:122] "RemoveContainer" containerID="ed869ac7baaa7c974728c827b04871d33eb723218f65a09c23937be5108b04c9" Oct 31 03:46:34.185712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.185693 4488 scope.go:122] "RemoveContainer" containerID="045335e4c30657ffe7549213646ea1ee7f5570cb70aa20501ad07707a9853192" Oct 31 03:46:34.195692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.195672 4488 scope.go:122] "RemoveContainer" containerID="96302356c75390148331f74ad2485a820515cb905ec0945ddd7416dab1632b77" Oct 31 03:46:34.204836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.204809 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:34.205564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.205540 4488 scope.go:122] "RemoveContainer" containerID="5f1d38281c6beff6fc3a808c7beb75131250b3738accb20c233f44db5c1573d2" Oct 31 03:46:34.216389 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.216370 4488 scope.go:122] "RemoveContainer" containerID="2a2bfa55ba7b861aeac331f654e32ec446fee0a9976643a91467844e3f128c35" Oct 31 03:46:34.229209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.229186 4488 scope.go:122] "RemoveContainer" containerID="d0f6db342c89f65e6489c6f9dadc57bb1401041ed51f9d48deb6c279ef820bfa" Oct 31 03:46:34.240357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.240333 4488 scope.go:122] "RemoveContainer" containerID="5e1bfdb39386fbb7d88165b6374549a33030e5ac6b69d8320c094124b90e625e" Oct 31 03:46:34.250615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.250595 4488 scope.go:122] "RemoveContainer" containerID="f5e6507135bd87f55ab362620c82238f118ff6a61eb5ebd0a7530abb95c6e5d4" Oct 31 03:46:34.260721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.260702 4488 scope.go:122] "RemoveContainer" containerID="9b4f59f76bd24850835a334fd365e5c1c6de5cea28b761bcc4485f24835ba6c5" Oct 31 03:46:34.320153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.320083 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c961755-b435-426b-a6aa-c26a3c479d5c" path="/var/lib/kubelet/pods/6c961755-b435-426b-a6aa-c26a3c479d5c/volumes" Oct 31 03:46:34.751273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.751241 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q"] Oct 31 03:46:34.788115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.782668 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.842987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.842954 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1180d818-openshift-default-dockercfg-gjfz4\"" Oct 31 03:46:34.859703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.859670 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:34.863799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.863763 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864149 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864331 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864447 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864544 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864727 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864832 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj4j\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.864936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.864924 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.865291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.865272 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.968395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.951189 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q"] Oct 31 03:46:34.970494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970438 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.970662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970501 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj4j\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.970662 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970532 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.970804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970752 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.970867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970852 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.970931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970887 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.970931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970920 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.971053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970965 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.971053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.970999 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.971675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.971647 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.973275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.972876 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.973275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.973184 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.974030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.973760 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.974030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.973979 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.991762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.991735 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:34.995348 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:34.995325 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:35.035727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.035654 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj4j\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:35.079195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.079151 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf"] Oct 31 03:46:35.084276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.084242 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33960ec2-k5ndf"] Oct 31 03:46:35.085153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.085037 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:35.125353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.125274 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:35.148442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.148406 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:35.618246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.618217 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q"] Oct 31 03:46:35.618963 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:46:35.618930 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d41da34_9b31_4d0d_a545_770e5a1d6e82.slice/crio-a858b86fbd8a3bfc241c22031df576cd6c03b543cf171e8b4c4e9d39ce33d1a4 WatchSource:0}: Error finding container a858b86fbd8a3bfc241c22031df576cd6c03b543cf171e8b4c4e9d39ce33d1a4: Status 404 returned error can't find the container with id a858b86fbd8a3bfc241c22031df576cd6c03b543cf171e8b4c4e9d39ce33d1a4 Oct 31 03:46:35.622308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.622268 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:46:35.622406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.622360 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:46:35.622449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:35.622407 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:46:36.259031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:36.258998 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" event={"ID":"7d41da34-9b31-4d0d-a545-770e5a1d6e82","Type":"ContainerStarted","Data":"14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b"} Oct 31 03:46:36.259447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:36.259050 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" event={"ID":"7d41da34-9b31-4d0d-a545-770e5a1d6e82","Type":"ContainerStarted","Data":"a858b86fbd8a3bfc241c22031df576cd6c03b543cf171e8b4c4e9d39ce33d1a4"} Oct 31 03:46:36.317955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:36.317921 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebfbb64-06bb-4cb2-aa80-8cb14f6138db" path="/var/lib/kubelet/pods/4ebfbb64-06bb-4cb2-aa80-8cb14f6138db/volumes" Oct 31 03:46:36.383360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:36.383312 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podStartSLOduration=2.383293471 podStartE2EDuration="2.383293471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:36.380171949 +0000 UTC m=+1624.176890210" watchObservedRunningTime="2026-10-31 03:46:36.383293471 +0000 UTC m=+1624.180011729" Oct 31 03:46:36.595992 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:36.595937 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:46:37.121009 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:37.120965 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:37.149640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:37.149609 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:37.175335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:37.175306 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:37.268935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:37.268670 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:37.271712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:37.271687 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:46:38.219922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:38.219882 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:46:39.233353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:39.233311 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:39.365839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:39.365808 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5"] Oct 31 03:46:39.449432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:39.449395 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:39.677399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:39.677369 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:39.838197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:39.834774 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:39.995829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:39.995740 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:42.052235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.052187 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx"] Oct 31 03:46:42.102309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.102273 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a0e4ab-rmnhx"] Oct 31 03:46:42.318431 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.318341 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed534e73-0c88-4c25-8dc9-b0a5e292c234" path="/var/lib/kubelet/pods/ed534e73-0c88-4c25-8dc9-b0a5e292c234/volumes" Oct 31 03:46:42.367595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.367563 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:42.947493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.947460 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp"] Oct 31 03:46:42.947719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.947698 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" containerID="cri-o://b3f99405a818b994ed7a5e384b3a975d9409e25c1eb2bb0c29df97fcf00ff840" gracePeriod=30 Oct 31 03:46:42.997821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.997785 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8"] Oct 31 03:46:42.998051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:42.998026 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" containerID="cri-o://cfdd3b7bccd1fa4bf9c4a8125697d305d5f68a661f3198f7d73251659ecbfefc" gracePeriod=30 Oct 31 03:46:43.001708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.001675 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph"] Oct 31 03:46:43.001931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.001909 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" containerID="cri-o://56a59a33bb8497ec045128cc24df4393dc4a8dff9df9d377f2581b4c7519771c" gracePeriod=30 Oct 31 03:46:43.048079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.048039 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl"] Oct 31 03:46:43.048340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.048272 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" containerID="cri-o://9b7ee91847064ceaad770c9d0cc2c8730bac4713b92d0c9fd6329ce4c872a8bd" gracePeriod=30 Oct 31 03:46:43.284728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.284644 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:43.496790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.496720 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:43.697343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.697312 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb"] Oct 31 03:46:43.706824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.706794 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:43.753021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.752977 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:43.753678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.753653 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwjz\" (UniqueName: \"kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:43.771127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.765057 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb"] Oct 31 03:46:43.771127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.767296 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1180d818\"" Oct 31 03:46:43.771127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.768112 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1180d818-dockercfg-46gbh\"" Oct 31 03:46:43.859109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.856759 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwjz\" (UniqueName: \"kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:43.859109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.856950 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:43.859109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.858011 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:43.977174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:43.975454 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwjz\" (UniqueName: \"kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:44.029933 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.029890 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:44.366146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.366109 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" podUID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" containerName="payload-processing" containerID="cri-o://5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059" gracePeriod=25 Oct 31 03:46:44.429059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.429019 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb"] Oct 31 03:46:44.450454 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:46:44.450410 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e2ddfc_6736_4776_b5f4_c3b7982106b5.slice/crio-3dc2d4d339a15b71a937db62a1065952dc6b0c71bfaf32c14752db9f03fcfc9d WatchSource:0}: Error finding container 3dc2d4d339a15b71a937db62a1065952dc6b0c71bfaf32c14752db9f03fcfc9d: Status 404 returned error can't find the container with id 3dc2d4d339a15b71a937db62a1065952dc6b0c71bfaf32c14752db9f03fcfc9d Oct 31 03:46:44.450617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:46:44.450467 4488 logging.go:55] [core] [Channel #1016 SubChannel #1017]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.135:50052", ServerName: "10.133.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.135:50052: connect: connection refused" Oct 31 03:46:44.647108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.647062 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:46:44.674381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.674352 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h\" (UniqueName: \"kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h\") pod \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " Oct 31 03:46:44.674638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.674555 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume\") pod \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\" (UID: \"e0c92668-f7f1-40ca-96d6-f3439daa54b3\") " Oct 31 03:46:44.674978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.674935 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume" pod "e0c92668-f7f1-40ca-96d6-f3439daa54b3" (UID: "e0c92668-f7f1-40ca-96d6-f3439daa54b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:44.675390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.675369 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c92668-f7f1-40ca-96d6-f3439daa54b3-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:44.677064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.677038 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h" pod "e0c92668-f7f1-40ca-96d6-f3439daa54b3" (UID: "e0c92668-f7f1-40ca-96d6-f3439daa54b3"). InnerVolumeSpecName "kube-api-access-phh8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:44.777607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:44.777574 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh8h\" (UniqueName: \"kubernetes.io/projected/e0c92668-f7f1-40ca-96d6-f3439daa54b3-kube-api-access-phh8h\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:45.337869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.337818 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" event={"ID":"89e2ddfc-6736-4776-b5f4-c3b7982106b5","Type":"ContainerStarted","Data":"77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a"} Oct 31 03:46:45.338067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.337886 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" event={"ID":"89e2ddfc-6736-4776-b5f4-c3b7982106b5","Type":"ContainerStarted","Data":"3dc2d4d339a15b71a937db62a1065952dc6b0c71bfaf32c14752db9f03fcfc9d"} Oct 31 03:46:45.338067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.337920 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:45.339288 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.339261 4488 generic.go:356] "Generic (PLEG): container finished" podID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" containerID="5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059" exitCode=0 Oct 31 03:46:45.339407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.339295 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" Oct 31 03:46:45.339407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.339336 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" event={"ID":"e0c92668-f7f1-40ca-96d6-f3439daa54b3","Type":"ContainerDied","Data":"5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059"} Oct 31 03:46:45.339407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.339380 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" event={"ID":"e0c92668-f7f1-40ca-96d6-f3439daa54b3","Type":"ContainerDied","Data":"36ef407ca06be670b4dcf7140eb4e253e8aa3cb48d0e33c6a28c687c610891d5"} Oct 31 03:46:45.339407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.339401 4488 scope.go:122] "RemoveContainer" containerID="5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059" Oct 31 03:46:45.351415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.351400 4488 scope.go:122] "RemoveContainer" containerID="5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059" Oct 31 03:46:45.351694 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:46:45.351675 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059\": container with ID starting with 5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059 not found: ID does not exist" containerID="5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059" Oct 31 03:46:45.351737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.351705 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059"} err="failed to get container status \"5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059\": rpc error: code = NotFound desc = could not find container \"5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059\": container with ID starting with 5bb7c1d926887bbd8ec867df7d8e9dc7691125f652de4561a15e876885ec5059 not found: ID does not exist" Oct 31 03:46:45.450375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.450327 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5" podUID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.135:50052\" within 1s: context deadline exceeded" Oct 31 03:46:45.457360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.457303 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" podStartSLOduration=2.457288387 podStartE2EDuration="2.457288387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:45.453929715 +0000 UTC m=+1633.250647975" watchObservedRunningTime="2026-10-31 03:46:45.457288387 +0000 UTC m=+1633.254006682" Oct 31 03:46:45.597077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.596177 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5"] Oct 31 03:46:45.598195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:45.598155 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb362142-799cfbd4df-bv9l5"] Oct 31 03:46:46.203137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:46.201762 4488 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:46:46.203137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:46.201825 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:46:46.318715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:46.318685 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" path="/var/lib/kubelet/pods/e0c92668-f7f1-40ca-96d6-f3439daa54b3/volumes" Oct 31 03:46:47.380064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:47.380022 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:48.766577 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.766547 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:46:48.770694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.770672 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:46:48.773819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.773786 4488 generic.go:356] "Generic (PLEG): container finished" podID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerID="56a59a33bb8497ec045128cc24df4393dc4a8dff9df9d377f2581b4c7519771c" exitCode=0 Oct 31 03:46:48.774282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.774255 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" event={"ID":"3b0e602d-d0bf-49fc-b13a-ea527237af0d","Type":"ContainerDied","Data":"56a59a33bb8497ec045128cc24df4393dc4a8dff9df9d377f2581b4c7519771c"} Oct 31 03:46:48.774460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.774446 4488 scope.go:122] "RemoveContainer" containerID="56a59a33bb8497ec045128cc24df4393dc4a8dff9df9d377f2581b4c7519771c" Oct 31 03:46:48.827661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.810257 4488 generic.go:356] "Generic (PLEG): container finished" podID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerID="9b7ee91847064ceaad770c9d0cc2c8730bac4713b92d0c9fd6329ce4c872a8bd" exitCode=0 Oct 31 03:46:48.827661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.810361 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" event={"ID":"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1","Type":"ContainerDied","Data":"9b7ee91847064ceaad770c9d0cc2c8730bac4713b92d0c9fd6329ce4c872a8bd"} Oct 31 03:46:48.827661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.822997 4488 generic.go:356] "Generic (PLEG): container finished" podID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerID="b3f99405a818b994ed7a5e384b3a975d9409e25c1eb2bb0c29df97fcf00ff840" exitCode=0 Oct 31 03:46:48.827661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.823165 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" event={"ID":"af6be095-5dfa-42f3-8cbe-fd404adcd57e","Type":"ContainerDied","Data":"b3f99405a818b994ed7a5e384b3a975d9409e25c1eb2bb0c29df97fcf00ff840"} Oct 31 03:46:48.827661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.823280 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp" Oct 31 03:46:48.858116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.857551 4488 generic.go:356] "Generic (PLEG): container finished" podID="0438525a-cdfc-4df1-9f73-701d38383aad" containerID="cfdd3b7bccd1fa4bf9c4a8125697d305d5f68a661f3198f7d73251659ecbfefc" exitCode=0 Oct 31 03:46:48.858116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.857650 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" event={"ID":"0438525a-cdfc-4df1-9f73-701d38383aad","Type":"ContainerDied","Data":"cfdd3b7bccd1fa4bf9c4a8125697d305d5f68a661f3198f7d73251659ecbfefc"} Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873167 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873221 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873273 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873362 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873425 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873466 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873504 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873534 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873704 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873741 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873787 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873815 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873871 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.873925 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.874064 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.874109 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.874204 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo\") pod \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\" (UID: \"3b0e602d-d0bf-49fc-b13a-ea527237af0d\") " Oct 31 03:46:48.875264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.874229 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo\") pod \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\" (UID: \"af6be095-5dfa-42f3-8cbe-fd404adcd57e\") " Oct 31 03:46:48.899971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.876755 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.899971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.885478 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.899971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.896849 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.899971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.897058 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.899971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.897409 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:48.902283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.901686 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:48.902283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.901943 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.913488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.913453 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.915225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.914903 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:48.977987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.964181 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989557 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989588 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989603 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989616 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989629 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989642 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989655 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989668 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989681 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.005625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:48.989694 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.112178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.111505 4488 scope.go:122] "RemoveContainer" containerID="aa7cf2c650ab5a4a840611bb3f2b5497518393d4e4a3e9b95b2f42415710d1d7" Oct 31 03:46:49.154116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.127791 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.206143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.200483 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.222622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.220333 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.222622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.220368 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.229529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.229473 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "kube-api-access-k77gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.234131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.234058 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:49.239791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.235268 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy" pod "3b0e602d-d0bf-49fc-b13a-ea527237af0d" (UID: "3b0e602d-d0bf-49fc-b13a-ea527237af0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.239791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.235768 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:49.239791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.239379 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.260414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.244427 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh" pod "af6be095-5dfa-42f3-8cbe-fd404adcd57e" (UID: "af6be095-5dfa-42f3-8cbe-fd404adcd57e"). InnerVolumeSpecName "kube-api-access-mdwvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.351928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.345846 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.351928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.345878 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.351928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.345895 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77gs\" (UniqueName: \"kubernetes.io/projected/3b0e602d-d0bf-49fc-b13a-ea527237af0d-kube-api-access-k77gs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.351928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.345910 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0e602d-d0bf-49fc-b13a-ea527237af0d-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.351928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.345923 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6be095-5dfa-42f3-8cbe-fd404adcd57e-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.351928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.345936 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwvh\" (UniqueName: \"kubernetes.io/projected/af6be095-5dfa-42f3-8cbe-fd404adcd57e-kube-api-access-mdwvh\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.406952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.406540 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:46:49.415621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.415212 4488 scope.go:122] "RemoveContainer" containerID="b3f99405a818b994ed7a5e384b3a975d9409e25c1eb2bb0c29df97fcf00ff840" Oct 31 03:46:49.446939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.446907 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447052 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447126 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447286 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.447372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447347 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447417 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447544 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447522 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447552 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:49.447667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447653 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447730 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.447813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447754 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.447813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.447806 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data\") pod \"0438525a-cdfc-4df1-9f73-701d38383aad\" (UID: \"0438525a-cdfc-4df1-9f73-701d38383aad\") " Oct 31 03:46:49.448753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.448726 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.448876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.448755 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0438525a-cdfc-4df1-9f73-701d38383aad-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.448876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.448773 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.449050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.449030 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.449556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.449527 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.452062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.452034 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "kube-api-access-nm7vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.452184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.452139 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.454404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.454384 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.455640 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.455614 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo" pod "0438525a-cdfc-4df1-9f73-701d38383aad" (UID: "0438525a-cdfc-4df1-9f73-701d38383aad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:49.467792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.467768 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:46:49.549881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.549846 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.550194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.549938 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.550194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.549978 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.550194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550023 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.550194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550179 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.550433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550177 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550770 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550853 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550851 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550900 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.550978 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token\") pod \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\" (UID: \"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1\") " Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.551457 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.551832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.551516 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.552862 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553070 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553112 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553126 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553137 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553141 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553162 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553208 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553223 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553241 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0438525a-cdfc-4df1-9f73-701d38383aad-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553312 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7vd\" (UniqueName: \"kubernetes.io/projected/0438525a-cdfc-4df1-9f73-701d38383aad-kube-api-access-nm7vd\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553330 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553360 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0438525a-cdfc-4df1-9f73-701d38383aad-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.553769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553750 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:49.553994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.553805 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "kube-api-access-mfsfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.555052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.555020 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token" pod "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" (UID: "eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:49.663619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.657122 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.663619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.657169 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfsfv\" (UniqueName: \"kubernetes.io/projected/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-kube-api-access-mfsfv\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.663619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.657183 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.663619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.657205 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:46:49.908757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.907448 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp"] Oct 31 03:46:49.927463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.927377 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ggfsp"] Oct 31 03:46:49.935678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.935620 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" event={"ID":"0438525a-cdfc-4df1-9f73-701d38383aad","Type":"ContainerDied","Data":"08856f05b434878a1d7eb57b48133192fdea4d01bad7cc123e5d3f4477f83c49"} Oct 31 03:46:49.935982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.935963 4488 scope.go:122] "RemoveContainer" containerID="cfdd3b7bccd1fa4bf9c4a8125697d305d5f68a661f3198f7d73251659ecbfefc" Oct 31 03:46:49.936277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.936260 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8" Oct 31 03:46:49.942366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.941768 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" event={"ID":"3b0e602d-d0bf-49fc-b13a-ea527237af0d","Type":"ContainerDied","Data":"6fe725a25fec9d2ef22c22e285002d50b92d9da6c0bc7023ace39c168805add6"} Oct 31 03:46:49.942366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.941869 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph" Oct 31 03:46:49.955618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.955583 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" event={"ID":"eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1","Type":"ContainerDied","Data":"9191f1a4312fbc031a870d21fa0e82a7156f08844cf4dcb1c13def419e513f27"} Oct 31 03:46:49.955984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:49.955889 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl" Oct 31 03:46:50.040386 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.040340 4488 scope.go:122] "RemoveContainer" containerID="9b7ee91847064ceaad770c9d0cc2c8730bac4713b92d0c9fd6329ce4c872a8bd" Oct 31 03:46:50.246251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.246150 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8"] Oct 31 03:46:50.265758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.265716 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-ks4m8"] Oct 31 03:46:50.365217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.365181 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" path="/var/lib/kubelet/pods/0438525a-cdfc-4df1-9f73-701d38383aad/volumes" Oct 31 03:46:50.366030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.366008 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" path="/var/lib/kubelet/pods/af6be095-5dfa-42f3-8cbe-fd404adcd57e/volumes" Oct 31 03:46:50.366506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.366486 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph"] Oct 31 03:46:50.366927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.366905 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-nq4ph"] Oct 31 03:46:50.489077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.489047 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl"] Oct 31 03:46:50.496509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:50.496429 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-pk8cl"] Oct 31 03:46:52.346112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:52.345567 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:52.392593 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:52.391959 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" path="/var/lib/kubelet/pods/3b0e602d-d0bf-49fc-b13a-ea527237af0d/volumes" Oct 31 03:46:52.392813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:52.392677 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" path="/var/lib/kubelet/pods/eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1/volumes" Oct 31 03:46:56.049282 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:56.049198 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx"] Oct 31 03:46:56.098796 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:56.098762 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e617087b-tshwx"] Oct 31 03:46:56.318325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:56.318239 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b267f797-7931-4277-9b98-baa6d8f9105a" path="/var/lib/kubelet/pods/b267f797-7931-4277-9b98-baa6d8f9105a/volumes" Oct 31 03:46:56.352077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:56.352050 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:46:57.090240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:57.090206 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:59.987997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:46:59.987939 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:00.145216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:00.145176 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:01.877237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:01.877193 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:02.038618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.038588 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:47:02.038817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.038649 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:47:02.129001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.128911 4488 generic.go:356] "Generic (PLEG): container finished" podID="0761e73a-f63e-4edc-958a-b96db8614d28" containerID="1528241113c33e48afc9d321875167590fd98ed89d2437a9dad6c4a597a4a8a1" exitCode=0 Oct 31 03:47:02.129261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.128993 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" event={"ID":"0761e73a-f63e-4edc-958a-b96db8614d28","Type":"ContainerDied","Data":"1528241113c33e48afc9d321875167590fd98ed89d2437a9dad6c4a597a4a8a1"} Oct 31 03:47:02.129707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.129684 4488 scope.go:122] "RemoveContainer" containerID="1528241113c33e48afc9d321875167590fd98ed89d2437a9dad6c4a597a4a8a1" Oct 31 03:47:02.131215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.131180 4488 generic.go:356] "Generic (PLEG): container finished" podID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerID="14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a" exitCode=0 Oct 31 03:47:02.131341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.131253 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" event={"ID":"ccf94248-c7de-465f-9f12-f9a0275f5a5d","Type":"ContainerDied","Data":"14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a"} Oct 31 03:47:02.131670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.131655 4488 scope.go:122] "RemoveContainer" containerID="14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a" Oct 31 03:47:02.131734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.131680 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:02.131802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.131780 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:02.131858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.131841 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:02.133375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.133346 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:02.133482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.133414 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:02.133482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:02.133450 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:03.164670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:03.164628 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" event={"ID":"0761e73a-f63e-4edc-958a-b96db8614d28","Type":"ContainerStarted","Data":"0a10fc13538260c4d15b7a7abb50fd45580b9bbb489964cc8eb97017d217568e"} Oct 31 03:47:03.167167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:03.167138 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" event={"ID":"ccf94248-c7de-465f-9f12-f9a0275f5a5d","Type":"ContainerStarted","Data":"20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907"} Oct 31 03:47:03.770004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:03.769970 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:47:03.771082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:03.771054 4488 patch_prober.go:29] interesting pod/e2e-mt-71186061-openshift-default-5698544949-klmg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 03:47:03.771298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:03.771277 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 03:47:04.026699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:04.026605 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:47:04.027058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:04.026966 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:47:04.027267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:04.027241 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:47:04.769615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:04.769581 4488 patch_prober.go:29] interesting pod/e2e-mt-71186061-openshift-default-5698544949-klmg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 03:47:04.770037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:04.769636 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 03:47:05.021196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:05.013542 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:47:05.021196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:05.013630 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:47:05.769965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:05.769925 4488 patch_prober.go:29] interesting pod/e2e-mt-71186061-openshift-default-5698544949-klmg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 03:47:05.770475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:05.769980 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 03:47:06.012116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:06.011527 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:47:06.012116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:06.011606 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:47:06.591149 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:47:06.591071 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:47:06.699856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:06.699822 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:06.780144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:06.775706 4488 patch_prober.go:29] interesting pod/e2e-mt-71186061-openshift-default-5698544949-klmg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 03:47:06.780144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:06.775775 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 03:47:07.025194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:07.021145 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:47:07.025194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:07.021222 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:47:07.116131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:07.113739 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:07.166509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:07.166470 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:47:07.769713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:07.769671 4488 patch_prober.go:29] interesting pod/e2e-mt-71186061-openshift-default-5698544949-klmg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 03:47:07.769910 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:07.769727 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 03:47:08.023184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:08.012544 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:47:08.023184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:08.012630 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 31 03:47:08.774396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:08.774364 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:47:08.775537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:08.775513 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:47:09.016672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:09.016641 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:47:09.017001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:09.016968 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:47:09.017873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:09.017855 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:47:11.459070 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:11.459032 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:13.914606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:13.914524 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq"] Oct 31 03:47:13.915014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:13.914737 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" containerID="cri-o://0747546007d747cef987b3f74770424a36bd625f31f055d053c9b4802d9f3019" gracePeriod=30 Oct 31 03:47:13.964995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:13.964952 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h"] Oct 31 03:47:13.965276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:13.965248 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" containerID="cri-o://29c4cdedae50c24f137830c086d0f0de0618bce9bcbd2f26e4acecd0dcc1afe2" gracePeriod=30 Oct 31 03:47:15.639971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:15.639920 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.148649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:17.148608 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.364443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:17.364409 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.616150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:17.616078 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.751374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:17.751338 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl"] Oct 31 03:47:17.785376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:17.785344 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb362142-zrgpl"] Oct 31 03:47:17.809153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:17.809068 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:18.318071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:18.318039 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5606f8f4-b409-401d-b30d-43726a2a109a" path="/var/lib/kubelet/pods/5606f8f4-b409-401d-b30d-43726a2a109a/volumes" Oct 31 03:47:18.634574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:18.634544 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:19.272523 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.272489 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p"] Oct 31 03:47:19.273701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.273673 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" Oct 31 03:47:19.273868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.273853 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" Oct 31 03:47:19.273979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.273968 4488 cpu_manager.go:414] "containerMap: removing container" podUID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" containerName="payload-processing" Oct 31 03:47:19.274054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274044 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" containerName="payload-processing" Oct 31 03:47:19.274215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274203 4488 cpu_manager.go:414] "containerMap: removing container" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" Oct 31 03:47:19.274291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274279 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" Oct 31 03:47:19.274637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274620 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" Oct 31 03:47:19.274750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274738 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" Oct 31 03:47:19.274864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274853 4488 cpu_manager.go:414] "containerMap: removing container" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" Oct 31 03:47:19.274949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.274937 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" Oct 31 03:47:19.275041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275030 4488 cpu_manager.go:414] "containerMap: removing container" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" Oct 31 03:47:19.275199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275184 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" Oct 31 03:47:19.275497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275474 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" Oct 31 03:47:19.275497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275502 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6be095-5dfa-42f3-8cbe-fd404adcd57e" containerName="istio-proxy" Oct 31 03:47:19.275708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275518 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0438525a-cdfc-4df1-9f73-701d38383aad" containerName="istio-proxy" Oct 31 03:47:19.275708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275531 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c92668-f7f1-40ca-96d6-f3439daa54b3" containerName="payload-processing" Oct 31 03:47:19.275708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.275548 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0e602d-d0bf-49fc-b13a-ea527237af0d" containerName="istio-proxy" Oct 31 03:47:19.280122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.280099 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.338245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.338209 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-36664b-openshift-default-dockercfg-bcwwp\"" Oct 31 03:47:19.365561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.365394 4488 generic.go:356] "Generic (PLEG): container finished" podID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerID="29c4cdedae50c24f137830c086d0f0de0618bce9bcbd2f26e4acecd0dcc1afe2" exitCode=0 Oct 31 03:47:19.365561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.365488 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" event={"ID":"3ace7f3c-d341-49a2-977f-580f8b3085df","Type":"ContainerDied","Data":"29c4cdedae50c24f137830c086d0f0de0618bce9bcbd2f26e4acecd0dcc1afe2"} Oct 31 03:47:19.367755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.367730 4488 generic.go:356] "Generic (PLEG): container finished" podID="066b1561-4bf0-49fc-835e-990f92496c8e" containerID="0747546007d747cef987b3f74770424a36bd625f31f055d053c9b4802d9f3019" exitCode=0 Oct 31 03:47:19.367991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.367961 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" event={"ID":"066b1561-4bf0-49fc-835e-990f92496c8e","Type":"ContainerDied","Data":"0747546007d747cef987b3f74770424a36bd625f31f055d053c9b4802d9f3019"} Oct 31 03:47:19.368200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.368183 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" event={"ID":"066b1561-4bf0-49fc-835e-990f92496c8e","Type":"ContainerDied","Data":"bc23c1763dd3e7dae7c2d2829c7b1bdd4b0644c97a3c44f2765d07d7cdfb3904"} Oct 31 03:47:19.368450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.368434 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc23c1763dd3e7dae7c2d2829c7b1bdd4b0644c97a3c44f2765d07d7cdfb3904" Oct 31 03:47:19.387726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.382793 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:19.392551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.392522 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p"] Oct 31 03:47:19.392753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.392731 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:47:19.397729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.397700 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422323 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422542 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422605 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422654 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422692 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422737 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422805 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9h96\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.426117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.422911 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.524665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524567 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.524834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524676 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.524834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524723 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.524834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524775 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.525026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524841 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524869 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524905 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.525026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524902 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.525026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.524972 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525025 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.525382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525066 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.525382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525153 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:19.525382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525198 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525262 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.525382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525290 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.525382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525308 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525409 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525439 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525452 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525455 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525470 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525519 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525556 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy\") pod \"066b1561-4bf0-49fc-835e-990f92496c8e\" (UID: \"066b1561-4bf0-49fc-835e-990f92496c8e\") " Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525580 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.525701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525588 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket\") pod \"3ace7f3c-d341-49a2-977f-580f8b3085df\" (UID: \"3ace7f3c-d341-49a2-977f-580f8b3085df\") " Oct 31 03:47:19.526255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525840 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525903 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:19.526255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526145 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.526420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.525907 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9h96\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526369 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.526420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526403 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526692 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526524 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526735 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526850 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526812 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526853 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.526908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526876 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.527034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526901 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.527034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526925 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.527034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.526977 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.527276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527212 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527234 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066b1561-4bf0-49fc-835e-990f92496c8e-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527249 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527265 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527278 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ace7f3c-d341-49a2-977f-580f8b3085df-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527292 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527307 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527320 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527335 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527348 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.527888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527628 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.527888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.527674 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.529463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.528360 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.530081 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.530540 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.530596 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "kube-api-access-ptxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.530770 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.531445 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token" pod "066b1561-4bf0-49fc-835e-990f92496c8e" (UID: "066b1561-4bf0-49fc-835e-990f92496c8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.531916 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.532310 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "kube-api-access-rvght". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:19.534146 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.533588 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy" pod "3ace7f3c-d341-49a2-977f-580f8b3085df" (UID: "3ace7f3c-d341-49a2-977f-580f8b3085df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:19.535257 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.534688 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.535257 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.534914 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.587580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.587549 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.587863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.587834 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9h96\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96\") pod \"e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:19.624785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.624753 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:19.629453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629427 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629457 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629473 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxq4\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-kube-api-access-ptxq4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629490 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066b1561-4bf0-49fc-835e-990f92496c8e-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629504 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ace7f3c-d341-49a2-977f-580f8b3085df-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629518 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvght\" (UniqueName: \"kubernetes.io/projected/3ace7f3c-d341-49a2-977f-580f8b3085df-kube-api-access-rvght\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629531 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066b1561-4bf0-49fc-835e-990f92496c8e-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.629563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.629544 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066b1561-4bf0-49fc-835e-990f92496c8e-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.688580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:19.688549 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:20.138147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.138119 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p"] Oct 31 03:47:20.139705 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:47:20.139682 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7b08bd_1c06_4308_84ed_f71ef4043463.slice/crio-604b5d5ef931ef7f373e82e81e40f60534fb86a7309e80f7bd97906957f178c7 WatchSource:0}: Error finding container 604b5d5ef931ef7f373e82e81e40f60534fb86a7309e80f7bd97906957f178c7: Status 404 returned error can't find the container with id 604b5d5ef931ef7f373e82e81e40f60534fb86a7309e80f7bd97906957f178c7 Oct 31 03:47:20.142607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.142565 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:20.142693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.142656 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:20.142742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.142700 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:20.379575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.379537 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" event={"ID":"4d7b08bd-1c06-4308-84ed-f71ef4043463","Type":"ContainerStarted","Data":"661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1"} Oct 31 03:47:20.380010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.379599 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" event={"ID":"4d7b08bd-1c06-4308-84ed-f71ef4043463","Type":"ContainerStarted","Data":"604b5d5ef931ef7f373e82e81e40f60534fb86a7309e80f7bd97906957f178c7"} Oct 31 03:47:20.381173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.381148 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" event={"ID":"3ace7f3c-d341-49a2-977f-580f8b3085df","Type":"ContainerDied","Data":"c0879608b9d3ef40dfd644b76f8499c56ec2654d78427de8926b968fa8342bb2"} Oct 31 03:47:20.381173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.381170 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h" Oct 31 03:47:20.381339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.381204 4488 scope.go:122] "RemoveContainer" containerID="29c4cdedae50c24f137830c086d0f0de0618bce9bcbd2f26e4acecd0dcc1afe2" Oct 31 03:47:20.381452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.381438 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq" Oct 31 03:47:20.505403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.505302 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podStartSLOduration=1.505286806 podStartE2EDuration="1.505286806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:20.501857276 +0000 UTC m=+1668.298575535" watchObservedRunningTime="2026-10-31 03:47:20.505286806 +0000 UTC m=+1668.302005064" Oct 31 03:47:20.640114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.636623 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq"] Oct 31 03:47:20.640114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.638289 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-mb2nq"] Oct 31 03:47:20.688789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.688759 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:20.733549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.733516 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h"] Oct 31 03:47:20.739683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:20.739654 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-svt6h"] Oct 31 03:47:21.696535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:21.696503 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:22.318124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:22.318070 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" path="/var/lib/kubelet/pods/066b1561-4bf0-49fc-835e-990f92496c8e/volumes" Oct 31 03:47:22.318776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:22.318755 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" path="/var/lib/kubelet/pods/3ace7f3c-d341-49a2-977f-580f8b3085df/volumes" Oct 31 03:47:22.399101 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:22.399051 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:22.400102 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:22.400064 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:47:22.674256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:22.674213 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:23.343292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:23.343259 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:23.507223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:23.507184 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:23.704942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:23.704852 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:26.386493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:26.386442 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:26.846963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:26.846928 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:27.254222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254132 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs"] Oct 31 03:47:27.254752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254733 4488 cpu_manager.go:414] "containerMap: removing container" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" Oct 31 03:47:27.254807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254755 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" Oct 31 03:47:27.254807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254784 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" Oct 31 03:47:27.254807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254789 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" Oct 31 03:47:27.254914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254904 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ace7f3c-d341-49a2-977f-580f8b3085df" containerName="istio-proxy" Oct 31 03:47:27.254949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254923 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb948593-f56f-46ec-b0a7-6ffa5c8c9dc1" containerName="istio-proxy" Oct 31 03:47:27.254949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.254932 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="066b1561-4bf0-49fc-835e-990f92496c8e" containerName="istio-proxy" Oct 31 03:47:27.257312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.257296 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.272693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.272661 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.272846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.272802 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.272846 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.272831 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2t82\" (UniqueName: \"kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.288956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.288926 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:47:27.290027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.290001 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:27.309915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.309882 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:27.310067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.310048 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:47:27.338776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.338740 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs"] Oct 31 03:47:27.374894 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.374840 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.375335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.374975 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.375335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.375009 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2t82\" (UniqueName: \"kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.375703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.375675 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.378918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.378885 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.437054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.437023 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2t82\" (UniqueName: \"kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82\") pod \"maas-api-revoke-sub-150d759e-xw6vs\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.570911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.570829 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:27.816777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.816740 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9"] Oct 31 03:47:27.820365 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.820344 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938558 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938650 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938699 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvsl\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938778 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938833 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938897 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.939112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.938986 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.943674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.942967 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.943674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.943052 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:27.945304 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.945276 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-2accc9-openshift-default-dockercfg-28clz\"" Oct 31 03:47:27.946804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.946742 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:27.979179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.978992 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9"] Oct 31 03:47:28.004062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:27.996488 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs"] Oct 31 03:47:28.044141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044102 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvsl\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044208 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044270 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044325 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044414 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044574 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044648 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044708 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044777 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044896 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.044985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.044961 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.045049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.045014 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.045294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.045273 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.046991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.046164 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.046991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.046966 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.047645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.047628 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.108891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.108801 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.112551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.112516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:28.205728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.205690 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvsl\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl\") pod \"e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.273521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.273489 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:28.441576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.441483 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv"] Oct 31 03:47:28.443167 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.443123 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:28.445325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.445301 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.452590 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.452560 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" event={"ID":"c76d79ee-398d-401c-859e-56030a5b97ad","Type":"ContainerStarted","Data":"2e792bf51a5b529e996725ce7b696f4ed083a7708aa4326d6c91b76203488f88"} Oct 31 03:47:28.502934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.500294 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-36664b-dockercfg-2wzjs\"" Oct 31 03:47:28.502934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.500379 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-36664b\"" Oct 31 03:47:28.502934 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.502480 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv"] Oct 31 03:47:28.575716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.575463 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.575936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.575745 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtjc\" (UniqueName: \"kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc\") pod \"payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.601218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.598144 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8"] Oct 31 03:47:28.611336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.609391 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:28.654485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.654432 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8"] Oct 31 03:47:28.677220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.677183 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtjc\" (UniqueName: \"kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc\") pod \"payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.677651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.677632 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.678783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.678756 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.744335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.744000 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtjc\" (UniqueName: \"kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc\") pod \"payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.827128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.802584 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:28.827128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.803856 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:28.827128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.804017 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tfg\" (UniqueName: \"kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg\") pod \"payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:28.910970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.906951 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tfg\" (UniqueName: \"kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg\") pod \"payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:28.910970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.908685 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:28.910970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.910674 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:28.917420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.917283 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9"] Oct 31 03:47:28.929879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.928433 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:28.930468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.930127 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:28.930468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.930180 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:47:28.972649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:28.972562 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tfg\" (UniqueName: \"kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg\") pod \"payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:29.240162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.240078 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:29.253296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.253264 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv"] Oct 31 03:47:29.257432 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:47:29.257331 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd52fc4_6175_4651_866f_c2afaec5c206.slice/crio-ed36f0597e7bc2a28a50a5a5cdc4a73c72b924db8e7993a8aca1648c0ecc085b WatchSource:0}: Error finding container ed36f0597e7bc2a28a50a5a5cdc4a73c72b924db8e7993a8aca1648c0ecc085b: Status 404 returned error can't find the container with id ed36f0597e7bc2a28a50a5a5cdc4a73c72b924db8e7993a8aca1648c0ecc085b Oct 31 03:47:29.462018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.461985 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" event={"ID":"fbd52fc4-6175-4651-866f-c2afaec5c206","Type":"ContainerStarted","Data":"473f0aebba9c127a172154e24d25ea18bab566693b6c1c668e906c01d10b315d"} Oct 31 03:47:29.462487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.462049 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" event={"ID":"fbd52fc4-6175-4651-866f-c2afaec5c206","Type":"ContainerStarted","Data":"ed36f0597e7bc2a28a50a5a5cdc4a73c72b924db8e7993a8aca1648c0ecc085b"} Oct 31 03:47:29.462487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.462216 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:29.464346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.464315 4488 generic.go:356] "Generic (PLEG): container finished" podID="c76d79ee-398d-401c-859e-56030a5b97ad" containerID="4682db2ca5c2526c6c7cd836a5a9c338fb76fa3d25cbf1b953cc3b185b2c179b" exitCode=0 Oct 31 03:47:29.464544 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.464454 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" event={"ID":"c76d79ee-398d-401c-859e-56030a5b97ad","Type":"ContainerDied","Data":"4682db2ca5c2526c6c7cd836a5a9c338fb76fa3d25cbf1b953cc3b185b2c179b"} Oct 31 03:47:29.466575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.466550 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" event={"ID":"0c937467-ba00-456f-a0cc-ef9f919855eb","Type":"ContainerStarted","Data":"e67379cc8282000c865d0b26b4c30314a0e7d833ff835595be5cdc1d6d8102fe"} Oct 31 03:47:29.466690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.466590 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" event={"ID":"0c937467-ba00-456f-a0cc-ef9f919855eb","Type":"ContainerStarted","Data":"c9cde71536ca45813c39032839b7c9ad460e70aca4e60c92cd21e56a69cd68cb"} Oct 31 03:47:29.582605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.582546 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" podStartSLOduration=1.582529037 podStartE2EDuration="1.582529037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:29.577642384 +0000 UTC m=+1677.374360642" watchObservedRunningTime="2026-10-31 03:47:29.582529037 +0000 UTC m=+1677.379247297" Oct 31 03:47:29.646966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.646936 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8"] Oct 31 03:47:29.648020 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:47:29.647988 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb932f75_50e8_43e1_b2d7_7d2b72e56146.slice/crio-a621f980a44e0d3a49e82e14f2f5fd1c98d8d344f9c718f1771fc4d1ef7667ba WatchSource:0}: Error finding container a621f980a44e0d3a49e82e14f2f5fd1c98d8d344f9c718f1771fc4d1ef7667ba: Status 404 returned error can't find the container with id a621f980a44e0d3a49e82e14f2f5fd1c98d8d344f9c718f1771fc4d1ef7667ba Oct 31 03:47:29.712766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:29.712700 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podStartSLOduration=2.712679843 podStartE2EDuration="2.712679843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:29.70187887 +0000 UTC m=+1677.498597129" watchObservedRunningTime="2026-10-31 03:47:29.712679843 +0000 UTC m=+1677.509398104" Oct 31 03:47:30.443552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.443518 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:30.450117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.450078 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:30.476758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.476723 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" event={"ID":"cb932f75-50e8-43e1-b2d7-7d2b72e56146","Type":"ContainerStarted","Data":"3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1"} Oct 31 03:47:30.477289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.476787 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" event={"ID":"cb932f75-50e8-43e1-b2d7-7d2b72e56146","Type":"ContainerStarted","Data":"a621f980a44e0d3a49e82e14f2f5fd1c98d8d344f9c718f1771fc4d1ef7667ba"} Oct 31 03:47:30.477493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.477473 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:30.477571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.477516 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:30.479119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.479081 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:47:30.645474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.645444 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:30.693741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.693624 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" podStartSLOduration=2.693607315 podStartE2EDuration="2.693607315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:30.690616928 +0000 UTC m=+1678.487335188" watchObservedRunningTime="2026-10-31 03:47:30.693607315 +0000 UTC m=+1678.490325572" Oct 31 03:47:30.738223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.738188 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca\") pod \"c76d79ee-398d-401c-859e-56030a5b97ad\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " Oct 31 03:47:30.738406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.738241 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82\" (UniqueName: \"kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82\") pod \"c76d79ee-398d-401c-859e-56030a5b97ad\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " Oct 31 03:47:30.738460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.738443 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca\") pod \"c76d79ee-398d-401c-859e-56030a5b97ad\" (UID: \"c76d79ee-398d-401c-859e-56030a5b97ad\") " Oct 31 03:47:30.738792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.738766 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca" pod "c76d79ee-398d-401c-859e-56030a5b97ad" (UID: "c76d79ee-398d-401c-859e-56030a5b97ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:30.739066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.739027 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:30.740605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.740580 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca" pod "c76d79ee-398d-401c-859e-56030a5b97ad" (UID: "c76d79ee-398d-401c-859e-56030a5b97ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:30.740728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.740622 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82" pod "c76d79ee-398d-401c-859e-56030a5b97ad" (UID: "c76d79ee-398d-401c-859e-56030a5b97ad"). InnerVolumeSpecName "kube-api-access-l2t82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:30.841592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.841557 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76d79ee-398d-401c-859e-56030a5b97ad-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:30.841898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:30.841850 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2t82\" (UniqueName: \"kubernetes.io/projected/c76d79ee-398d-401c-859e-56030a5b97ad-kube-api-access-l2t82\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:31.486566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:31.486538 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" Oct 31 03:47:31.486980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:31.486564 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs" event={"ID":"c76d79ee-398d-401c-859e-56030a5b97ad","Type":"ContainerDied","Data":"2e792bf51a5b529e996725ce7b696f4ed083a7708aa4326d6c91b76203488f88"} Oct 31 03:47:31.486980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:31.486620 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e792bf51a5b529e996725ce7b696f4ed083a7708aa4326d6c91b76203488f88" Oct 31 03:47:32.609930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:32.609893 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:32.796010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:32.795983 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:32.961446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:32.961366 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:33.283584 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:33.283498 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:34.373438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.373403 4488 scope.go:122] "RemoveContainer" containerID="ee7d500ef553a0f9d8d67c755b9694e4d3a5803acba13ef4000dc5c27bad900d" Oct 31 03:47:34.384240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.384214 4488 scope.go:122] "RemoveContainer" containerID="fc5cc877450bcb0f4ffc57d85f0f355303e092fd37f1ceea419ea48be2829692" Oct 31 03:47:34.394639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.394619 4488 scope.go:122] "RemoveContainer" containerID="9dd2ccc44aa4c52e937ebb8daef96e24a2f61c9510a2e130ded6dbc369cee65d" Oct 31 03:47:34.404405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.404385 4488 scope.go:122] "RemoveContainer" containerID="e9993bd242b0dc51ad087e233a06de3a06b797a2c69f4ac808452cdc1ab269e6" Oct 31 03:47:34.414283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.414258 4488 scope.go:122] "RemoveContainer" containerID="dc1ddf85efa0ddcc8e13d73142268ee2d611198a3f1dffd4533f7302698fae41" Oct 31 03:47:34.424477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.424457 4488 scope.go:122] "RemoveContainer" containerID="14cd78d81496b2824aff6b1c653e96f70a6e3e2cd779e2e2bb6db6fd192879ed" Oct 31 03:47:34.434865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.434843 4488 scope.go:122] "RemoveContainer" containerID="47691a27f673e21d2416c3e59bc4fd38dbda580f627e1cf3a31f3c85f9acd9b6" Oct 31 03:47:34.444997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.444976 4488 scope.go:122] "RemoveContainer" containerID="0747546007d747cef987b3f74770424a36bd625f31f055d053c9b4802d9f3019" Oct 31 03:47:34.455465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.455445 4488 scope.go:122] "RemoveContainer" containerID="1e18b41febb6a01ddc3a707e590d2eeba153ac52f538a777839b8abee63dc0da" Oct 31 03:47:34.465372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.465353 4488 scope.go:122] "RemoveContainer" containerID="16cb9ab86b439d087ee1274ff68acf42f5914ffc36e85ee158f27ebb96c6ee7d" Oct 31 03:47:34.475918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.475889 4488 scope.go:122] "RemoveContainer" containerID="6a21c37e4eb7a9142350491b2e440456072eba2552bc6610294b9662d9af22f2" Oct 31 03:47:34.486059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.486035 4488 scope.go:122] "RemoveContainer" containerID="1ce047eb6c0e5322c5ef2ceddeced77d59d78ba8d993e94164faa82920a18c77" Oct 31 03:47:34.496530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.496508 4488 scope.go:122] "RemoveContainer" containerID="210b4b750414606ef602718d4f3461e135109e380d93886047ee3af5038a46d3" Oct 31 03:47:34.513385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.513360 4488 scope.go:122] "RemoveContainer" containerID="74b8432956d7c0ae6c016b4db036f450b0b2cdafb210fbc91b9082f260ddeefc" Oct 31 03:47:34.525745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.525717 4488 scope.go:122] "RemoveContainer" containerID="21ff372bac3a5a7ddf7dd1180eb7b02c88db14d9e5f9d747693eec7ed2832a86" Oct 31 03:47:34.538157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:34.538135 4488 scope.go:122] "RemoveContainer" containerID="b936d09c0d0a58c5a58150f2bf09c17fbf9ce2d801178488ba980f5ec4b0903b" Oct 31 03:47:36.437189 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:36.437158 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:36.622833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:36.619188 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:36.685587 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:47:36.684316 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:47:37.196129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:37.180380 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:38.170234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:38.170201 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:38.323966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:38.323916 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:38.484532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:38.484441 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:38.689848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:38.689807 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:40.479837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:40.479808 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:47:41.490341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:41.490305 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:47:43.244388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.244351 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6"] Oct 31 03:47:43.245019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.245005 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c76d79ee-398d-401c-859e-56030a5b97ad" containerName="revoke-keys" Oct 31 03:47:43.245065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.245020 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76d79ee-398d-401c-859e-56030a5b97ad" containerName="revoke-keys" Oct 31 03:47:43.245176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.245166 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76d79ee-398d-401c-859e-56030a5b97ad" containerName="revoke-keys" Oct 31 03:47:43.249571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.249550 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.321555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.321520 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5vp6\" (UniqueName: \"kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.321794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.321763 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.321876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.321814 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.321967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.321950 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.352777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.352737 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1180d818-serving-cert\"" Oct 31 03:47:43.356842 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.356811 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6"] Oct 31 03:47:43.423074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.423043 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5vp6\" (UniqueName: \"kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.423277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.423248 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.423277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.423273 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.423422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.423406 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.423922 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:47:43.423871 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-serving-cert: secret "maas-api-e2e-mt-1180d818-serving-cert" not found Oct 31 03:47:43.425326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.424717 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.425326 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:47:43.425231 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls podName:64d269d0-254a-4a11-8f72-68c8a060deb7 nodeName:}" failed. No retries permitted until 2026-10-31 03:47:43.925207658 +0000 UTC m=+1691.721925894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls") pod "maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" (UID: "64d269d0-254a-4a11-8f72-68c8a060deb7") : secret "maas-api-e2e-mt-1180d818-serving-cert" not found Oct 31 03:47:43.437826 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.437791 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.455666 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.455634 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:43.485588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.485554 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5vp6\" (UniqueName: \"kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.650054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.650020 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:43.811821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.811785 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:43.939793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.939694 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:43.942447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:43.942419 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:44.163671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:44.163638 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:44.548971 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:47:44.548936 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d269d0_254a_4a11_8f72_68c8a060deb7.slice/crio-c7a70d18bacebcb62fcf1d027ade54c02b84a0ebc27c6c7fd7d5934b6881bc1d WatchSource:0}: Error finding container c7a70d18bacebcb62fcf1d027ade54c02b84a0ebc27c6c7fd7d5934b6881bc1d: Status 404 returned error can't find the container with id c7a70d18bacebcb62fcf1d027ade54c02b84a0ebc27c6c7fd7d5934b6881bc1d Oct 31 03:47:44.549425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:44.548993 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6"] Oct 31 03:47:44.683147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:44.683074 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" event={"ID":"64d269d0-254a-4a11-8f72-68c8a060deb7","Type":"ContainerStarted","Data":"c7a70d18bacebcb62fcf1d027ade54c02b84a0ebc27c6c7fd7d5934b6881bc1d"} Oct 31 03:47:44.947370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:44.947305 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:45.318371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:45.318293 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:45.470816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:45.470784 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:45.628030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:45.627997 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:45.692106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:45.692064 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" event={"ID":"64d269d0-254a-4a11-8f72-68c8a060deb7","Type":"ContainerStarted","Data":"f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67"} Oct 31 03:47:45.692319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:45.692147 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:45.813690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:45.813626 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" podStartSLOduration=2.002355908 podStartE2EDuration="2.813612135s" totalImagesPullingTime="811.256227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:45.810409069 +0000 UTC m=+1693.607127323" watchObservedRunningTime="2026-10-31 03:47:45.813612135 +0000 UTC m=+1693.610330423" Oct 31 03:47:46.104347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.104312 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc"] Oct 31 03:47:46.107376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.107359 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.190650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.190610 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.190955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.190932 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwsb\" (UniqueName: \"kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.191261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.191237 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.192345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.192321 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc"] Oct 31 03:47:46.194257 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.194238 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:46.214528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.214484 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:47:46.293245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.293210 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.293472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.293257 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwsb\" (UniqueName: \"kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.293472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.293335 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.294186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.294162 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.295988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.295962 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.354944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.354864 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwsb\" (UniqueName: \"kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb\") pod \"maas-api-revoke-keys-e2e-mt-71186061-6d6sc\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.420546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.420510 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:46.805481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:46.805454 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc"] Oct 31 03:47:46.806741 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:47:46.806715 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3f5370_643e_476e_9fb4_4062579a4a19.slice/crio-145a6ade12585b0f848bbb0d50498ee088cdb9b7f867fa34937d589a38bfcecb WatchSource:0}: Error finding container 145a6ade12585b0f848bbb0d50498ee088cdb9b7f867fa34937d589a38bfcecb: Status 404 returned error can't find the container with id 145a6ade12585b0f848bbb0d50498ee088cdb9b7f867fa34937d589a38bfcecb Oct 31 03:47:47.712537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:47.712497 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" event={"ID":"be3f5370-643e-476e-9fb4-4062579a4a19","Type":"ContainerStarted","Data":"e32a9bc563fbf2a52c47d4300359bfd97a4b6bc9eb09fde4a9a034fd6525caa3"} Oct 31 03:47:47.712832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:47.712803 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" event={"ID":"be3f5370-643e-476e-9fb4-4062579a4a19","Type":"ContainerStarted","Data":"145a6ade12585b0f848bbb0d50498ee088cdb9b7f867fa34937d589a38bfcecb"} Oct 31 03:47:47.836036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:47.835975 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" podStartSLOduration=1.8359578779999999 podStartE2EDuration="1.835957878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:47.835644047 +0000 UTC m=+1695.632362322" watchObservedRunningTime="2026-10-31 03:47:47.835957878 +0000 UTC m=+1695.632676137" Oct 31 03:47:48.722238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:48.722203 4488 generic.go:356] "Generic (PLEG): container finished" podID="be3f5370-643e-476e-9fb4-4062579a4a19" containerID="e32a9bc563fbf2a52c47d4300359bfd97a4b6bc9eb09fde4a9a034fd6525caa3" exitCode=0 Oct 31 03:47:48.722456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:48.722285 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" event={"ID":"be3f5370-643e-476e-9fb4-4062579a4a19","Type":"ContainerDied","Data":"e32a9bc563fbf2a52c47d4300359bfd97a4b6bc9eb09fde4a9a034fd6525caa3"} Oct 31 03:47:48.917521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:48.917487 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:49.899658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:49.899630 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:50.024579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.024476 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca\") pod \"be3f5370-643e-476e-9fb4-4062579a4a19\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " Oct 31 03:47:50.025063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.024636 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca\") pod \"be3f5370-643e-476e-9fb4-4062579a4a19\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " Oct 31 03:47:50.025063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.024765 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb\" (UniqueName: \"kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb\") pod \"be3f5370-643e-476e-9fb4-4062579a4a19\" (UID: \"be3f5370-643e-476e-9fb4-4062579a4a19\") " Oct 31 03:47:50.025063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.024989 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca" pod "be3f5370-643e-476e-9fb4-4062579a4a19" (UID: "be3f5370-643e-476e-9fb4-4062579a4a19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:50.025549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.025523 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:50.027876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.027847 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb" pod "be3f5370-643e-476e-9fb4-4062579a4a19" (UID: "be3f5370-643e-476e-9fb4-4062579a4a19"). InnerVolumeSpecName "kube-api-access-tzwsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:50.028172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.028147 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca" pod "be3f5370-643e-476e-9fb4-4062579a4a19" (UID: "be3f5370-643e-476e-9fb4-4062579a4a19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:50.127058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.127018 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3f5370-643e-476e-9fb4-4062579a4a19-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:50.127058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.127048 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwsb\" (UniqueName: \"kubernetes.io/projected/be3f5370-643e-476e-9fb4-4062579a4a19-kube-api-access-tzwsb\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:47:50.739570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.739479 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" event={"ID":"be3f5370-643e-476e-9fb4-4062579a4a19","Type":"ContainerDied","Data":"145a6ade12585b0f848bbb0d50498ee088cdb9b7f867fa34937d589a38bfcecb"} Oct 31 03:47:50.739570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.739515 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc" Oct 31 03:47:50.739570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:50.739531 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="145a6ade12585b0f848bbb0d50498ee088cdb9b7f867fa34937d589a38bfcecb" Oct 31 03:47:51.708284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:51.708254 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:47:51.991313 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:51.991228 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:54.149255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:54.149223 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:57.967804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:57.967729 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:58.131680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:58.131644 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:59.018635 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.018603 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:59.663995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.663959 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z"] Oct 31 03:47:59.664223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.664200 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" containerID="cri-o://6b2cb37cdbc97d8851ffd0de4ab0d297c2309e67b10b147372215acd99a7bb16" gracePeriod=30 Oct 31 03:47:59.949181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.949025 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl"] Oct 31 03:47:59.949346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.949313 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" containerID="cri-o://9d5fec12ff09556a3e4e54dccf945ce4eb5ddac0439134a71163dd9337869899" gracePeriod=30 Oct 31 03:47:59.952651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.952619 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49"] Oct 31 03:47:59.952891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.952853 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" containerID="cri-o://6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4" gracePeriod=30 Oct 31 03:47:59.956317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.956291 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9"] Oct 31 03:47:59.956491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:47:59.956472 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" containerID="cri-o://cccdfc70a2829abc1c6a7195f898a1c1e1c7921a113e984298044bc389525b88" gracePeriod=30 Oct 31 03:48:03.590978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:03.590941 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:03.591491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:03.591013 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:03.833540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:03.833504 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:04.851493 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:04.851452 4488 generic.go:356] "Generic (PLEG): container finished" podID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerID="6b2cb37cdbc97d8851ffd0de4ab0d297c2309e67b10b147372215acd99a7bb16" exitCode=0 Oct 31 03:48:04.852068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:04.851654 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" event={"ID":"42c6c416-ff4e-4d46-a1d8-8d188260e9ee","Type":"ContainerDied","Data":"6b2cb37cdbc97d8851ffd0de4ab0d297c2309e67b10b147372215acd99a7bb16"} Oct 31 03:48:05.097249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.077563 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:05.217931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.214222 4488 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:48:05.217931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.214280 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:48:05.241192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.240316 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358268 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358331 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358432 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358566 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358615 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358693 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358769 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358832 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.358936 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert\") pod \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\" (UID: \"42c6c416-ff4e-4d46-a1d8-8d188260e9ee\") " Oct 31 03:48:05.360738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.360410 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.360738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.360564 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.362050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.361951 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.362050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.362021 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.363181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.362940 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:05.363362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.363323 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:05.376691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.376633 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "kube-api-access-z88dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:05.377126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.377071 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.395247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.394785 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:05.395247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.395202 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo" pod "42c6c416-ff4e-4d46-a1d8-8d188260e9ee" (UID: "42c6c416-ff4e-4d46-a1d8-8d188260e9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466503 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466539 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466558 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466572 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466586 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88dz\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-kube-api-access-z88dz\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466600 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466614 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466626 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.502287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.466639 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c6c416-ff4e-4d46-a1d8-8d188260e9ee-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.665158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.665069 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.770688 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.770747 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.770837 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.770995 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.771046 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.771116 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.771297 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.771328 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.772419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.771358 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket\") pod \"1a86723a-a51e-4e8d-b121-bc831aaf1456\" (UID: \"1a86723a-a51e-4e8d-b121-bc831aaf1456\") " Oct 31 03:48:05.773237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.773198 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.842080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.841702 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:05.843962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.843749 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.850163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.844185 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.867553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.854227 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.904117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.894606 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.904117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.894675 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a86723a-a51e-4e8d-b121-bc831aaf1456-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.904117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.894704 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.904117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.894720 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.904117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.894735 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.930388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.916619 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.930388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.920645 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "kube-api-access-2l7mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:05.930388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.921364 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:05.960471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:05.934828 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo" pod "1a86723a-a51e-4e8d-b121-bc831aaf1456" (UID: "1a86723a-a51e-4e8d-b121-bc831aaf1456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:06.014443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.012453 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l7mz\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-kube-api-access-2l7mz\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.014443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.012489 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.014443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.012509 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.014443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.012531 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a86723a-a51e-4e8d-b121-bc831aaf1456-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.029343 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" event={"ID":"42c6c416-ff4e-4d46-a1d8-8d188260e9ee","Type":"ContainerDied","Data":"1d9e66aa5835342b8310ace09d22c96f604766973c1312059bf41d5837eafa1e"} Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.029458 4488 scope.go:122] "RemoveContainer" containerID="6b2cb37cdbc97d8851ffd0de4ab0d297c2309e67b10b147372215acd99a7bb16" Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.029625 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z" Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.043337 4488 generic.go:356] "Generic (PLEG): container finished" podID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerID="9d5fec12ff09556a3e4e54dccf945ce4eb5ddac0439134a71163dd9337869899" exitCode=0 Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.043498 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" event={"ID":"3d2f9169-43a4-4aa2-920f-33774916eb13","Type":"ContainerDied","Data":"9d5fec12ff09556a3e4e54dccf945ce4eb5ddac0439134a71163dd9337869899"} Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.046673 4488 generic.go:356] "Generic (PLEG): container finished" podID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerID="cccdfc70a2829abc1c6a7195f898a1c1e1c7921a113e984298044bc389525b88" exitCode=0 Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.047066 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" event={"ID":"726d0934-3c43-46dc-8b27-be9a0dc2ac71","Type":"ContainerDied","Data":"cccdfc70a2829abc1c6a7195f898a1c1e1c7921a113e984298044bc389525b88"} Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.052162 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" event={"ID":"1a86723a-a51e-4e8d-b121-bc831aaf1456","Type":"ContainerDied","Data":"6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4"} Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.051667 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" Oct 31 03:48:06.052533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.051576 4488 generic.go:356] "Generic (PLEG): container finished" podID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerID="6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4" exitCode=0 Oct 31 03:48:06.068788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.057152 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49" event={"ID":"1a86723a-a51e-4e8d-b121-bc831aaf1456","Type":"ContainerDied","Data":"44edd9bb868caa875586b8f4d2a655bc567ef161fcbed479544ed97ff0759b01"} Oct 31 03:48:06.135676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.135542 4488 scope.go:122] "RemoveContainer" containerID="6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4" Oct 31 03:48:06.366715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.350575 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:48:06.369053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.367904 4488 scope.go:122] "RemoveContainer" containerID="6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4" Oct 31 03:48:06.369053 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:06.368935 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4\": container with ID starting with 6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4 not found: ID does not exist" containerID="6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4" Oct 31 03:48:06.369053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.368974 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4"} err="failed to get container status \"6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4\": rpc error: code = NotFound desc = could not find container \"6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4\": container with ID starting with 6149fc34e89116c67c0d7be49a919626999570a12bbeb62430a81efb552138b4 not found: ID does not exist" Oct 31 03:48:06.444949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.444904 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.445649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.445515 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.445873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.445840 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.446204 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.446076 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.466828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.448014 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z"] Oct 31 03:48:06.468499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.457211 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.468499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.466616 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.468940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.468910 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-clr4z"] Oct 31 03:48:06.469138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.466784 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.469554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.469535 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.470082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.470061 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.470399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.470358 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.471081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.471058 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.476584 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.476546 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token\") pod \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\" (UID: \"726d0934-3c43-46dc-8b27-be9a0dc2ac71\") " Oct 31 03:48:06.478317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.469694 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49"] Oct 31 03:48:06.478501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.478486 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-hgf49"] Oct 31 03:48:06.478603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.471295 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.478690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.471525 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:06.483353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.482961 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.483353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.482997 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.483353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.483014 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.483353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.483029 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.483353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.483044 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.483353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.483269 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:48:06.490402 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.490340 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "kube-api-access-fnpjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:06.507145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.504390 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:06.519626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.519569 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.520222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.520196 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token" pod "726d0934-3c43-46dc-8b27-be9a0dc2ac71" (UID: "726d0934-3c43-46dc-8b27-be9a0dc2ac71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:06.637678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.623721 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.637678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.623761 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnpjn\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-kube-api-access-fnpjn\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.637678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.623778 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.637678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.623791 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d0934-3c43-46dc-8b27-be9a0dc2ac71-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728250 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728330 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728375 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728467 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728506 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728569 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728604 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728668 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.728855 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt\") pod \"3d2f9169-43a4-4aa2-920f-33774916eb13\" (UID: \"3d2f9169-43a4-4aa2-920f-33774916eb13\") " Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.777230 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.777509 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.789265 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.803193 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:06.879343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.803966 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:07.038178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.956575 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.038178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.956621 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.038178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.956637 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.038178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.956653 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.038178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.956667 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2f9169-43a4-4aa2-920f-33774916eb13-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.038178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:06.979615 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:07.142160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.070186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.142160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.095716 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2f9169-43a4-4aa2-920f-33774916eb13-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.215281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.161987 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.215281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.171196 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.215281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.193450 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:07.215281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.197584 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.236523 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.236406 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "kube-api-access-bjjqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:07.287147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.284372 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" event={"ID":"726d0934-3c43-46dc-8b27-be9a0dc2ac71","Type":"ContainerDied","Data":"c13d75ccc27179a5c097f73d1e712b2566c43d53599cb876c9f16ba89edbd107"} Oct 31 03:48:07.287147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.284451 4488 scope.go:122] "RemoveContainer" containerID="cccdfc70a2829abc1c6a7195f898a1c1e1c7921a113e984298044bc389525b88" Oct 31 03:48:07.287147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.284591 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9" Oct 31 03:48:07.322164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.299478 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjqt\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-kube-api-access-bjjqt\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.347119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.346071 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" event={"ID":"3d2f9169-43a4-4aa2-920f-33774916eb13","Type":"ContainerDied","Data":"ee09bc4db61cf2c8a8f53834a7edd22982eff77beb925d62bc7b86e9c2826be6"} Oct 31 03:48:07.371114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.346214 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl" Oct 31 03:48:07.371114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.369460 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token" pod "3d2f9169-43a4-4aa2-920f-33774916eb13" (UID: "3d2f9169-43a4-4aa2-920f-33774916eb13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:07.449121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.444965 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2f9169-43a4-4aa2-920f-33774916eb13-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:07.474449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.472276 4488 scope.go:122] "RemoveContainer" containerID="9d5fec12ff09556a3e4e54dccf945ce4eb5ddac0439134a71163dd9337869899" Oct 31 03:48:07.474449 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:07.472423 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:48:07.474449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.472545 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:48:07.602214 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.602145 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9"] Oct 31 03:48:07.653118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.651527 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-d6sp9"] Oct 31 03:48:07.745455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.744236 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.793461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.792196 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.812116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.807829 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:48:07.833130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.831236 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.833130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.831702 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 31 03:48:07.847132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.846187 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.898588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.898142 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl"] Oct 31 03:48:07.934115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.933740 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.934329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.934166 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.938115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.934476 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:48:07.981267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.981210 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:07.981571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.981541 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:48:07.984639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:07.984604 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-wrcbl"] Oct 31 03:48:08.055251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.055201 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:48:08.074252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.074192 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.075555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.075519 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.076236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.076212 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:48:08.141294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.141186 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.392176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.351321 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.491260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.452288 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kw8rf" podUID="af3162c5-fd5c-49bc-97f6-0897c174cc11" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.17:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.500616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.500572 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:48:08.588116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.576162 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.699250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.698008 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" path="/var/lib/kubelet/pods/1a86723a-a51e-4e8d-b121-bc831aaf1456/volumes" Oct 31 03:48:08.720439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.720391 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" path="/var/lib/kubelet/pods/3d2f9169-43a4-4aa2-920f-33774916eb13/volumes" Oct 31 03:48:08.721705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.721678 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" path="/var/lib/kubelet/pods/42c6c416-ff4e-4d46-a1d8-8d188260e9ee/volumes" Oct 31 03:48:08.723327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.723289 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" path="/var/lib/kubelet/pods/726d0934-3c43-46dc-8b27-be9a0dc2ac71/volumes" Oct 31 03:48:08.724145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.724127 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:48:08.750783 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.750740 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:48:08.751634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.751607 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:48:08.797247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.797200 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.799026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.798998 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:48:08.810499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.810459 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 31 03:48:08.811374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.811349 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:48:08.838924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.835632 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.838924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.836255 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:48:08.862119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.858621 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:08.862119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:08.859021 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:48:09.181733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.181700 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:09.186586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.186172 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:09.187340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.187317 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:48:09.195106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.188037 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:09.195401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.195367 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:48:09.195911 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.195875 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:48:09.197265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.196080 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:09.197265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.196898 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:48:09.222235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.197978 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:48:09.222235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.198859 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:48:09.222235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.215623 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 31 03:48:09.222235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.216023 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:09.222235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.216637 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:48:09.223657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.223624 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 31 03:48:09.224151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.224126 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 03:48:09.224558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.224538 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 03:48:09.258116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.233330 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:48:09.264264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.264237 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:48:09.264488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.264477 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:48:09.298193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.271152 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:48:09.298193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.271222 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:48:09.298193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.276712 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:48:09.401111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.382695 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:48:09.408122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.401448 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:48:09.487579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.487492 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 03:48:09.580123 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.578860 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:48:09.590114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.589084 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:48:09.610124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.602118 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:48:09.662360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.644033 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:09.670947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.645425 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 03:48:09.673504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.673245 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:48:09.673504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.673314 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:48:09.673504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.673367 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:48:09.673504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.673419 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 03:48:09.757228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:09.757144 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:48:13.802555 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:13.802519 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:14.800060 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:14.800021 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv"] Oct 31 03:48:14.800410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:14.800341 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" containerID="cri-o://2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95" gracePeriod=30 Oct 31 03:48:15.147766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:15.147722 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8"] Oct 31 03:48:15.148404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:15.148007 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" containerID="cri-o://bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa" gracePeriod=30 Oct 31 03:48:15.153969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:15.153928 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w"] Oct 31 03:48:15.154469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:15.154436 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" containerID="cri-o://eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9" gracePeriod=30 Oct 31 03:48:15.160068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:15.160035 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7"] Oct 31 03:48:15.160355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:15.160323 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" containerID="cri-o://6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365" gracePeriod=30 Oct 31 03:48:16.149937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:16.149900 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:16.447677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:16.447642 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:17.265790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.265694 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:17.831599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.831565 4488 generic.go:356] "Generic (PLEG): container finished" podID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerID="14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b" exitCode=0 Oct 31 03:48:17.831926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.831637 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" event={"ID":"7d41da34-9b31-4d0d-a545-770e5a1d6e82","Type":"ContainerDied","Data":"14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b"} Oct 31 03:48:17.832252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.832234 4488 scope.go:122] "RemoveContainer" containerID="14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b" Oct 31 03:48:17.834165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.834134 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:48:17.834266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.834207 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:48:17.834266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.834237 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:48:17.931033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.930996 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:17.931224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:17.931067 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:18.591360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:18.591326 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:18.850533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:18.850418 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" event={"ID":"7d41da34-9b31-4d0d-a545-770e5a1d6e82","Type":"ContainerStarted","Data":"5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206"} Oct 31 03:48:19.149756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:19.149653 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:48:19.150952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:19.150924 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:48:19.151127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:19.150992 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:48:19.858526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:19.858495 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:19.858892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:19.858556 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:20.137793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.137758 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:48:20.149459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.149415 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:48:20.149732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.149493 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:48:20.274263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274057 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274263 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274241 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274278 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274417 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274436 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:20.274569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274450 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274480 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274540 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.274569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274552 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274724 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.274810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274778 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.274914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274857 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.274970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.274950 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy\") pod \"4effa134-50cc-4790-b168-e246266aeaf1\" (UID: \"4effa134-50cc-4790-b168-e246266aeaf1\") " Oct 31 03:48:20.275889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.275364 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.275889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.275810 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.275889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.275833 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.275889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.275849 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.275889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.275860 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.275889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.275869 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4effa134-50cc-4790-b168-e246266aeaf1-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.279512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.279474 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.279832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.279810 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "kube-api-access-kjxpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.293358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.293295 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:20.293358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.293331 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token" pod "4effa134-50cc-4790-b168-e246266aeaf1" (UID: "4effa134-50cc-4790-b168-e246266aeaf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.378656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.377547 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxpp\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-kube-api-access-kjxpp\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.378656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.377583 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4effa134-50cc-4790-b168-e246266aeaf1-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.378656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.377599 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4effa134-50cc-4790-b168-e246266aeaf1-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.378656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.377608 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4effa134-50cc-4790-b168-e246266aeaf1-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.455841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.453378 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:48:20.455841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.453442 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:48:20.539989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.539926 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:48:20.579480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579447 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579519 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579585 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579639 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579717 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579834 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579889 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.579985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.579959 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.580134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.580048 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs\") pod \"55945c52-6050-46ea-98ad-3a561451409a\" (UID: \"55945c52-6050-46ea-98ad-3a561451409a\") " Oct 31 03:48:20.580492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.580464 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.580833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.580811 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.581001 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.580977 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.584076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.584050 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.585226 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.585202 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.585355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.585227 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "kube-api-access-g6kn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.585449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.585410 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.586103 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.586063 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:20.587106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.587062 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.600065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.600024 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo" pod "55945c52-6050-46ea-98ad-3a561451409a" (UID: "55945c52-6050-46ea-98ad-3a561451409a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:20.645124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.645081 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:48:20.677625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.677367 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:48:20.686476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686445 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.686628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686497 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.686628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686561 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.686628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686586 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.686628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686603 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.686628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686627 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.686792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686660 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.686792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686698 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.686792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686746 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.686887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686796 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.686887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686811 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.686887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686841 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.686887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686864 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.687012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686888 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.687012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686920 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.687012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686966 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data\") pod \"5d365752-8a68-4d24-9b3d-ff4413476da4\" (UID: \"5d365752-8a68-4d24-9b3d-ff4413476da4\") " Oct 31 03:48:20.687012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.686986 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.687158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687015 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo\") pod \"6b225622-1fce-4260-981d-cc845888d525\" (UID: \"6b225622-1fce-4260-981d-cc845888d525\") " Oct 31 03:48:20.687519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687504 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687521 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687530 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687540 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6kn4\" (UniqueName: \"kubernetes.io/projected/55945c52-6050-46ea-98ad-3a561451409a-kube-api-access-g6kn4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687550 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945c52-6050-46ea-98ad-3a561451409a-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687559 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945c52-6050-46ea-98ad-3a561451409a-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687566 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.687597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.687576 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945c52-6050-46ea-98ad-3a561451409a-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.688598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.688565 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:20.688955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.688918 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.689206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.689181 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.690781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.689453 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:20.690781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.689601 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.690947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.689777 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:20.690947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.689801 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.690947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.690006 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.690947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.690021 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.690947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.690391 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.690947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.690511 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.692427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.692398 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.692539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.692435 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "kube-api-access-f7m78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.693241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.693217 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:20.693342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.693244 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.693342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.693316 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "kube-api-access-ktqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.693463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.693442 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo" pod "5d365752-8a68-4d24-9b3d-ff4413476da4" (UID: "5d365752-8a68-4d24-9b3d-ff4413476da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:20.694458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.694433 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token" pod "6b225622-1fce-4260-981d-cc845888d525" (UID: "6b225622-1fce-4260-981d-cc845888d525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:20.789260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789220 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7m78\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-kube-api-access-f7m78\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789252 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b225622-1fce-4260-981d-cc845888d525-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789263 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789273 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789281 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d365752-8a68-4d24-9b3d-ff4413476da4-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789290 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789298 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789306 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789313 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789323 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b225622-1fce-4260-981d-cc845888d525-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789332 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789340 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b225622-1fce-4260-981d-cc845888d525-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789349 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789357 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqz2\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-kube-api-access-ktqz2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789365 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b225622-1fce-4260-981d-cc845888d525-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789374 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789381 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.789521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.789389 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d365752-8a68-4d24-9b3d-ff4413476da4-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:20.880817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.880774 4488 generic.go:356] "Generic (PLEG): container finished" podID="55945c52-6050-46ea-98ad-3a561451409a" containerID="bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa" exitCode=0 Oct 31 03:48:20.881367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.880966 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" Oct 31 03:48:20.881367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.881273 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" event={"ID":"55945c52-6050-46ea-98ad-3a561451409a","Type":"ContainerDied","Data":"bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa"} Oct 31 03:48:20.881501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.881396 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8" event={"ID":"55945c52-6050-46ea-98ad-3a561451409a","Type":"ContainerDied","Data":"a647f5c94878a0657ab597f3e49933bac4aa0009ecf86d8e42fbdcbb3df5bc34"} Oct 31 03:48:20.881501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.881436 4488 scope.go:122] "RemoveContainer" containerID="bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa" Oct 31 03:48:20.883524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.883492 4488 generic.go:356] "Generic (PLEG): container finished" podID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerID="6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365" exitCode=0 Oct 31 03:48:20.883653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.883633 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" event={"ID":"5d365752-8a68-4d24-9b3d-ff4413476da4","Type":"ContainerDied","Data":"6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365"} Oct 31 03:48:20.883763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.883687 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" event={"ID":"5d365752-8a68-4d24-9b3d-ff4413476da4","Type":"ContainerDied","Data":"b924ee0b8cb469a4e49ee62d4a5442e71f00c43fa1ff6398e69360fac8bf5e78"} Oct 31 03:48:20.883763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.883763 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7" Oct 31 03:48:20.889337 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.889275 4488 generic.go:356] "Generic (PLEG): container finished" podID="4effa134-50cc-4790-b168-e246266aeaf1" containerID="2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95" exitCode=0 Oct 31 03:48:20.889484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.889468 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" Oct 31 03:48:20.889955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.889465 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" event={"ID":"4effa134-50cc-4790-b168-e246266aeaf1","Type":"ContainerDied","Data":"2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95"} Oct 31 03:48:20.890047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.889989 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv" event={"ID":"4effa134-50cc-4790-b168-e246266aeaf1","Type":"ContainerDied","Data":"c3eb419f41a031f3129aa45ae5012f4dd45a5436dd6f347612b1cd0d3eb4bec5"} Oct 31 03:48:20.892753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.892579 4488 generic.go:356] "Generic (PLEG): container finished" podID="6b225622-1fce-4260-981d-cc845888d525" containerID="eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9" exitCode=0 Oct 31 03:48:20.892753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.892647 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" Oct 31 03:48:20.892753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.892684 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" event={"ID":"6b225622-1fce-4260-981d-cc845888d525","Type":"ContainerDied","Data":"eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9"} Oct 31 03:48:20.892753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.892721 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w" event={"ID":"6b225622-1fce-4260-981d-cc845888d525","Type":"ContainerDied","Data":"bb0a757e48aefeb40423ab5fa62fd9b8724203717afaaf82851410ac43a69695"} Oct 31 03:48:20.895659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.895629 4488 scope.go:122] "RemoveContainer" containerID="bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa" Oct 31 03:48:20.896060 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:20.895983 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa\": container with ID starting with bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa not found: ID does not exist" containerID="bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa" Oct 31 03:48:20.896060 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.896038 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa"} err="failed to get container status \"bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa\": rpc error: code = NotFound desc = could not find container \"bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa\": container with ID starting with bd84a32c52593c4b704c8392d1370196c48715d717607e57805fb238368f18fa not found: ID does not exist" Oct 31 03:48:20.896305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.896065 4488 scope.go:122] "RemoveContainer" containerID="6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365" Oct 31 03:48:20.941847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.941686 4488 scope.go:122] "RemoveContainer" containerID="6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365" Oct 31 03:48:20.942143 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:20.942120 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365\": container with ID starting with 6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365 not found: ID does not exist" containerID="6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365" Oct 31 03:48:20.942225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.942156 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365"} err="failed to get container status \"6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365\": rpc error: code = NotFound desc = could not find container \"6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365\": container with ID starting with 6a4d8f3c90bccf9fef509d3c72e8dd34c99a6857f042adffa52f174579550365 not found: ID does not exist" Oct 31 03:48:20.942225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.942181 4488 scope.go:122] "RemoveContainer" containerID="2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95" Oct 31 03:48:20.959220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.959188 4488 scope.go:122] "RemoveContainer" containerID="2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95" Oct 31 03:48:20.959621 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:20.959596 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95\": container with ID starting with 2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95 not found: ID does not exist" containerID="2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95" Oct 31 03:48:20.959718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.959630 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95"} err="failed to get container status \"2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95\": rpc error: code = NotFound desc = could not find container \"2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95\": container with ID starting with 2e27a00223c4b0ea4b169473d7540b6f8a1bbf8a0ef262f9cb50fcb204256b95 not found: ID does not exist" Oct 31 03:48:20.959718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.959661 4488 scope.go:122] "RemoveContainer" containerID="eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9" Oct 31 03:48:20.971849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.971828 4488 scope.go:122] "RemoveContainer" containerID="eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9" Oct 31 03:48:20.972317 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:20.972293 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9\": container with ID starting with eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9 not found: ID does not exist" containerID="eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9" Oct 31 03:48:20.972429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:20.972363 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9"} err="failed to get container status \"eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9\": rpc error: code = NotFound desc = could not find container \"eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9\": container with ID starting with eece9c49f3b82adfdffe2ff5945de424013679639dfedcb2c91e5b438086b5b9 not found: ID does not exist" Oct 31 03:48:21.063338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.063298 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv"] Oct 31 03:48:21.069278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.069249 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-phkmv"] Oct 31 03:48:21.149990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.149894 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:48:21.149990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.149968 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:48:21.182731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.182695 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7"] Oct 31 03:48:21.187597 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.187562 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-qrnn7"] Oct 31 03:48:21.298380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.298341 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8"] Oct 31 03:48:21.303454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.303420 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-x84w8"] Oct 31 03:48:21.433049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.432933 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w"] Oct 31 03:48:21.436452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:21.436418 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-lvf8w"] Oct 31 03:48:22.151960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.151855 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:48:22.152553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.152057 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:48:22.268973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.268933 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:48:22.340904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.340872 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4effa134-50cc-4790-b168-e246266aeaf1" path="/var/lib/kubelet/pods/4effa134-50cc-4790-b168-e246266aeaf1/volumes" Oct 31 03:48:22.341681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.341641 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55945c52-6050-46ea-98ad-3a561451409a" path="/var/lib/kubelet/pods/55945c52-6050-46ea-98ad-3a561451409a/volumes" Oct 31 03:48:22.342206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.342191 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" path="/var/lib/kubelet/pods/5d365752-8a68-4d24-9b3d-ff4413476da4/volumes" Oct 31 03:48:22.342953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:22.342939 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b225622-1fce-4260-981d-cc845888d525" path="/var/lib/kubelet/pods/6b225622-1fce-4260-981d-cc845888d525/volumes" Oct 31 03:48:23.150413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:23.150379 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:48:23.150600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:23.150460 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:48:23.399876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:23.399846 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:24.150301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:24.150262 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:48:24.150467 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:24.150327 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:48:25.154066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:25.154037 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:48:25.155267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:25.155239 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:48:26.774695 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:26.774610 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:27.953407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.953371 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx"] Oct 31 03:48:27.954159 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954138 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954159 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954176 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954183 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954203 4488 cpu_manager.go:414] "containerMap: removing container" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954210 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954228 4488 cpu_manager.go:414] "containerMap: removing container" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954234 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954252 4488 cpu_manager.go:414] "containerMap: removing container" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954259 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954269 4488 cpu_manager.go:414] "containerMap: removing container" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" Oct 31 03:48:27.954279 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954276 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954319 4488 cpu_manager.go:414] "containerMap: removing container" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954326 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954336 4488 cpu_manager.go:414] "containerMap: removing container" podUID="be3f5370-643e-476e-9fb4-4062579a4a19" containerName="revoke-keys" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954343 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3f5370-643e-476e-9fb4-4062579a4a19" containerName="revoke-keys" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954352 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954358 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954508 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="55945c52-6050-46ea-98ad-3a561451409a" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954522 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d365752-8a68-4d24-9b3d-ff4413476da4" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954537 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="726d0934-3c43-46dc-8b27-be9a0dc2ac71" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954548 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3f5370-643e-476e-9fb4-4062579a4a19" containerName="revoke-keys" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954559 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b225622-1fce-4260-981d-cc845888d525" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954570 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4effa134-50cc-4790-b168-e246266aeaf1" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954579 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2f9169-43a4-4aa2-920f-33774916eb13" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954588 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a86723a-a51e-4e8d-b121-bc831aaf1456" containerName="istio-proxy" Oct 31 03:48:27.954795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.954598 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c6c416-ff4e-4d46-a1d8-8d188260e9ee" containerName="istio-proxy" Oct 31 03:48:27.957261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:27.957242 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.009669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.009633 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-36664b-serving-cert\"" Oct 31 03:48:28.020768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.020730 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.020975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.020843 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzsnc\" (UniqueName: \"kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.020975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.020914 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.021181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.021162 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.063438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.063404 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx"] Oct 31 03:48:28.122758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.122722 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsnc\" (UniqueName: \"kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.122939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.122790 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.122939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.122925 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.123016 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.122956 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.123111 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:28.123069 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-serving-cert: secret "maas-api-e2e-shared-a-w6-36664b-serving-cert" not found Oct 31 03:48:28.123175 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:28.123164 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls podName:4193e587-c7b7-497e-95c5-3327ac467cfd nodeName:}" failed. No retries permitted until 2026-10-31 03:48:28.623146106 +0000 UTC m=+1736.419864360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls") pod "maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" (UID: "4193e587-c7b7-497e-95c5-3327ac467cfd") : secret "maas-api-e2e-shared-a-w6-36664b-serving-cert" not found Oct 31 03:48:28.123571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.123545 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.125417 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.125395 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.149218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.149176 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:28.186632 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.186598 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsnc\" (UniqueName: \"kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.311482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.311378 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:28.474029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.473982 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:28.631332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.631296 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.633960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.633934 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:28.870998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:28.870958 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:29.254058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:29.254032 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx"] Oct 31 03:48:29.255489 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:48:29.255464 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4193e587_c7b7_497e_95c5_3327ac467cfd.slice/crio-f8f701e98f594b2123ac20cbf8f863ef4bde83b1e286341aa253bb0d84cb4539 WatchSource:0}: Error finding container f8f701e98f594b2123ac20cbf8f863ef4bde83b1e286341aa253bb0d84cb4539: Status 404 returned error can't find the container with id f8f701e98f594b2123ac20cbf8f863ef4bde83b1e286341aa253bb0d84cb4539 Oct 31 03:48:29.991154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:29.991071 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" event={"ID":"4193e587-c7b7-497e-95c5-3327ac467cfd","Type":"ContainerStarted","Data":"f8f701e98f594b2123ac20cbf8f863ef4bde83b1e286341aa253bb0d84cb4539"} Oct 31 03:48:31.024494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:31.024457 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" event={"ID":"4193e587-c7b7-497e-95c5-3327ac467cfd","Type":"ContainerStarted","Data":"22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765"} Oct 31 03:48:31.024967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:31.024548 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:31.145250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:31.145196 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" podStartSLOduration=3.315021096 podStartE2EDuration="4.145181506s" totalImagesPullingTime="830.16041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:31.14242077 +0000 UTC m=+1738.939139029" watchObservedRunningTime="2026-10-31 03:48:31.145181506 +0000 UTC m=+1738.941899763" Oct 31 03:48:33.085180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:33.085126 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:34.692688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:34.692650 4488 scope.go:122] "RemoveContainer" containerID="5df63d5fd39083fe0062a6b4fc062b764dc846644c1897f00eb39e08258c1890" Oct 31 03:48:34.703807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:34.703779 4488 scope.go:122] "RemoveContainer" containerID="4906a461893d4af21597e63875a495dcb33b6568105dcc4f1c228f47b99c0b7d" Oct 31 03:48:36.589896 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:36.589856 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:48:37.036925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.036850 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:48:37.212066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.212032 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f"] Oct 31 03:48:37.212298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.212277 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" podUID="190d507f-0545-429c-9ee6-9c7f7c081b66" containerName="maas-api" containerID="cri-o://e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b" gracePeriod=30 Oct 31 03:48:37.320333 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.320251 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:37.517394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.517369 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:48:37.668328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.668297 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca\") pod \"190d507f-0545-429c-9ee6-9c7f7c081b66\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " Oct 31 03:48:37.668775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.668424 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\") pod \"190d507f-0545-429c-9ee6-9c7f7c081b66\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " Oct 31 03:48:37.668775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.668633 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw\" (UniqueName: \"kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw\") pod \"190d507f-0545-429c-9ee6-9c7f7c081b66\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " Oct 31 03:48:37.668775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.668688 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs\") pod \"190d507f-0545-429c-9ee6-9c7f7c081b66\" (UID: \"190d507f-0545-429c-9ee6-9c7f7c081b66\") " Oct 31 03:48:37.668775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.668691 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca" pod "190d507f-0545-429c-9ee6-9c7f7c081b66" (UID: "190d507f-0545-429c-9ee6-9c7f7c081b66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:37.669293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.669266 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190d507f-0545-429c-9ee6-9c7f7c081b66-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.671423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.671392 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw" pod "190d507f-0545-429c-9ee6-9c7f7c081b66" (UID: "190d507f-0545-429c-9ee6-9c7f7c081b66"). InnerVolumeSpecName "kube-api-access-lgmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:37.671611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.671591 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs" pod "190d507f-0545-429c-9ee6-9c7f7c081b66" (UID: "190d507f-0545-429c-9ee6-9c7f7c081b66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:37.674130 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.674067 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls" pod "190d507f-0545-429c-9ee6-9c7f7c081b66" (UID: "190d507f-0545-429c-9ee6-9c7f7c081b66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:48:37.771117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.770751 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmjw\" (UniqueName: \"kubernetes.io/projected/190d507f-0545-429c-9ee6-9c7f7c081b66-kube-api-access-lgmjw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.771117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.770792 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.771117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.770828 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190d507f-0545-429c-9ee6-9c7f7c081b66-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.922860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:37.922730 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:38.085957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.085923 4488 generic.go:356] "Generic (PLEG): container finished" podID="190d507f-0545-429c-9ee6-9c7f7c081b66" containerID="e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b" exitCode=0 Oct 31 03:48:38.086154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.085974 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" Oct 31 03:48:38.086154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.086009 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" event={"ID":"190d507f-0545-429c-9ee6-9c7f7c081b66","Type":"ContainerDied","Data":"e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b"} Oct 31 03:48:38.086154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.086064 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f" event={"ID":"190d507f-0545-429c-9ee6-9c7f7c081b66","Type":"ContainerDied","Data":"8d85cea182d0fd6e8d5b80d8c30803bebef9cb5b42c4edef42daf09f6d0e79e0"} Oct 31 03:48:38.086154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.086131 4488 scope.go:122] "RemoveContainer" containerID="e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b" Oct 31 03:48:38.107496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.107468 4488 scope.go:122] "RemoveContainer" containerID="e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b" Oct 31 03:48:38.107822 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:38.107797 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b\": container with ID starting with e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b not found: ID does not exist" containerID="e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b" Oct 31 03:48:38.107904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.107830 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b"} err="failed to get container status \"e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b\": rpc error: code = NotFound desc = could not find container \"e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b\": container with ID starting with e6b5f9c9b9371241b4917fc8eb3df06e3c49960cde0850106757c0b74ae3299b not found: ID does not exist" Oct 31 03:48:38.278728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.278611 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f"] Oct 31 03:48:38.326548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.326507 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71186061-774c54847b-8kb7f"] Oct 31 03:48:38.775173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.775140 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw"] Oct 31 03:48:38.775701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.775688 4488 cpu_manager.go:414] "containerMap: removing container" podUID="190d507f-0545-429c-9ee6-9c7f7c081b66" containerName="maas-api" Oct 31 03:48:38.775701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.775701 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190d507f-0545-429c-9ee6-9c7f7c081b66" containerName="maas-api" Oct 31 03:48:38.775829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.775825 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="190d507f-0545-429c-9ee6-9c7f7c081b66" containerName="maas-api" Oct 31 03:48:38.778763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.778745 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.790451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.790424 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.790623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.790605 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqsj\" (UniqueName: \"kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.790676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.790662 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.790761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.790744 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.831289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.831261 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-2accc9-serving-cert\"" Oct 31 03:48:38.839565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.839529 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw"] Oct 31 03:48:38.860743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.860711 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:38.892191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.892159 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.892370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.892311 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqsj\" (UniqueName: \"kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.892409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.892379 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.892441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.892427 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.892533 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:38.892515 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-serving-cert: secret "maas-api-e2e-shared-b-w1-2accc9-serving-cert" not found Oct 31 03:48:38.892613 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:48:38.892590 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls podName:f88f2d72-7161-447d-9b3e-0731e2448c48 nodeName:}" failed. No retries permitted until 2026-10-31 03:48:39.392573717 +0000 UTC m=+1747.189291955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls") pod "maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" (UID: "f88f2d72-7161-447d-9b3e-0731e2448c48") : secret "maas-api-e2e-shared-b-w1-2accc9-serving-cert" not found Oct 31 03:48:38.893134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.893078 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.894984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.894963 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:38.955452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:38.955419 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqsj\" (UniqueName: \"kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:39.032212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.026523 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:39.195704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.195634 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:39.398024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.397988 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:39.400795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.400769 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:39.433987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.433943 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc"] Oct 31 03:48:39.437144 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.437109 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71186061-6d6sc"] Oct 31 03:48:39.441206 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.441179 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:39.692074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.691975 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:39.810076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:39.810020 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:40.126734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:40.126703 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw"] Oct 31 03:48:40.128019 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:48:40.127985 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f2d72_7161_447d_9b3e_0731e2448c48.slice/crio-11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8 WatchSource:0}: Error finding container 11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8: Status 404 returned error can't find the container with id 11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8 Oct 31 03:48:40.318873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:40.318843 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190d507f-0545-429c-9ee6-9c7f7c081b66" path="/var/lib/kubelet/pods/190d507f-0545-429c-9ee6-9c7f7c081b66/volumes" Oct 31 03:48:40.319376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:40.319360 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3f5370-643e-476e-9fb4-4062579a4a19" path="/var/lib/kubelet/pods/be3f5370-643e-476e-9fb4-4062579a4a19/volumes" Oct 31 03:48:41.109297 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:41.109264 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" event={"ID":"f88f2d72-7161-447d-9b3e-0731e2448c48","Type":"ContainerStarted","Data":"d0b8f525d811152edf08e1f6bb2cbabe3f4b52e67538fd652a3195bcf56f9f4a"} Oct 31 03:48:41.109699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:41.109315 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" event={"ID":"f88f2d72-7161-447d-9b3e-0731e2448c48","Type":"ContainerStarted","Data":"11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8"} Oct 31 03:48:41.109699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:41.109498 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:41.228256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:41.228149 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" podStartSLOduration=2.462811576 podStartE2EDuration="3.228131791s" totalImagesPullingTime="765.320215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:41.227212468 +0000 UTC m=+1749.023930725" watchObservedRunningTime="2026-10-31 03:48:41.228131791 +0000 UTC m=+1749.024850050" Oct 31 03:48:42.052198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:42.052156 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h"] Oct 31 03:48:42.100299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:42.100266 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db2ec9e-plt2h"] Oct 31 03:48:42.317403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:42.317324 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e7f54a-3a9d-4720-b6d8-dfed1b2674be" path="/var/lib/kubelet/pods/d0e7f54a-3a9d-4720-b6d8-dfed1b2674be/volumes" Oct 31 03:48:44.217938 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:44.217853 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:44.888961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:44.888923 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:46.601244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:46.601202 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:46.803244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:46.803212 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:46.941561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:46.941453 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5"] Oct 31 03:48:46.941758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:46.941734 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" containerID="cri-o://0a10fc13538260c4d15b7a7abb50fd45580b9bbb489964cc8eb97017d217568e" gracePeriod=30 Oct 31 03:48:46.979799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:46.979768 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:47.122686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:47.122656 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:48:47.770218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:47.770190 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:48.337494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.337453 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs"] Oct 31 03:48:48.341856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.341831 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.394381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.394343 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs"] Oct 31 03:48:48.400143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.400074 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:48.439414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.439383 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bc91a3-openshift-default-dockercfg-9dfdn\"" Oct 31 03:48:48.450313 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450275 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450327 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450355 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450425 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450454 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75hwf\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450519 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450586 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450648 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.450727 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.450691 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552484 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552580 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552645 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552697 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552736 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552761 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552790 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.552841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552829 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75hwf\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.553010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.552925 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.553321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.553291 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.553321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.553306 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.553510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.553365 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.553510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.553422 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.553949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.553927 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.555380 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.555360 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.555499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.555474 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.652170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.652057 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.653546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.653515 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75hwf\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf\") pod \"e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.664822 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.664791 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:48.885546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:48.885509 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:49.133487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:49.133458 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs"] Oct 31 03:48:49.134572 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:48:49.134551 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8114c66e_0643_42d4_9ee0_e9c9d1923852.slice/crio-ae2d2deed260019cc8ad10646643c754cdedaf8ec097fcb1b6402ea0b8addcbb WatchSource:0}: Error finding container ae2d2deed260019cc8ad10646643c754cdedaf8ec097fcb1b6402ea0b8addcbb: Status 404 returned error can't find the container with id ae2d2deed260019cc8ad10646643c754cdedaf8ec097fcb1b6402ea0b8addcbb Oct 31 03:48:49.137399 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:49.137364 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:48:49.137473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:49.137434 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:48:49.137473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:49.137462 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:48:49.167376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:49.167346 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" event={"ID":"8114c66e-0643-42d4-9ee0-e9c9d1923852","Type":"ContainerStarted","Data":"ae2d2deed260019cc8ad10646643c754cdedaf8ec097fcb1b6402ea0b8addcbb"} Oct 31 03:48:49.285072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:49.285027 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:50.189866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:50.189805 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" event={"ID":"8114c66e-0643-42d4-9ee0-e9c9d1923852","Type":"ContainerStarted","Data":"57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2"} Oct 31 03:48:50.367231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:50.367166 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podStartSLOduration=2.367144424 podStartE2EDuration="2.367144424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:50.364264225 +0000 UTC m=+1758.160982507" watchObservedRunningTime="2026-10-31 03:48:50.367144424 +0000 UTC m=+1758.163862684" Oct 31 03:48:50.666050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:50.666011 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:50.674071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:50.674043 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:51.196923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:51.196884 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:51.198071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:51.198052 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:48:52.206483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.206412 4488 generic.go:356] "Generic (PLEG): container finished" podID="0761e73a-f63e-4edc-958a-b96db8614d28" containerID="0a10fc13538260c4d15b7a7abb50fd45580b9bbb489964cc8eb97017d217568e" exitCode=0 Oct 31 03:48:52.206829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.206494 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" event={"ID":"0761e73a-f63e-4edc-958a-b96db8614d28","Type":"ContainerDied","Data":"0a10fc13538260c4d15b7a7abb50fd45580b9bbb489964cc8eb97017d217568e"} Oct 31 03:48:52.206829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.206570 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" event={"ID":"0761e73a-f63e-4edc-958a-b96db8614d28","Type":"ContainerDied","Data":"1b5b76e37275901ab449eb28c60f5a5546a9a7e0139c32c2d623994a1692d72a"} Oct 31 03:48:52.206829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.206592 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b5b76e37275901ab449eb28c60f5a5546a9a7e0139c32c2d623994a1692d72a" Oct 31 03:48:52.206829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.206610 4488 scope.go:122] "RemoveContainer" containerID="1528241113c33e48afc9d321875167590fd98ed89d2437a9dad6c4a597a4a8a1" Oct 31 03:48:52.226658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.226635 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:48:52.322352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322323 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322381 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322412 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322451 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322491 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322524 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322623 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322673 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.322802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322681 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:52.322802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322731 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:52.322802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322740 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:52.323063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322835 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:52.323063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.322894 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4\") pod \"0761e73a-f63e-4edc-958a-b96db8614d28\" (UID: \"0761e73a-f63e-4edc-958a-b96db8614d28\") " Oct 31 03:48:52.323063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.323050 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:52.323811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.323781 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.323961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.323886 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.323961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.323920 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.323961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.323950 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.324111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.323963 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0761e73a-f63e-4edc-958a-b96db8614d28-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.325035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.325005 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:52.325376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.325359 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "kube-api-access-m6hf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:52.325534 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.325520 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:52.325624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.325602 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo" pod "0761e73a-f63e-4edc-958a-b96db8614d28" (UID: "0761e73a-f63e-4edc-958a-b96db8614d28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:52.425738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.425699 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hf4\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-kube-api-access-m6hf4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.425738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.425729 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0761e73a-f63e-4edc-958a-b96db8614d28-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.425738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.425745 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0761e73a-f63e-4edc-958a-b96db8614d28-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.425948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:52.425756 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0761e73a-f63e-4edc-958a-b96db8614d28-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:48:53.215217 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:53.215189 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5" Oct 31 03:48:53.235740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:53.235712 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:53.413449 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:53.413400 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:53.514792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:53.514694 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5"] Oct 31 03:48:53.519037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:53.519005 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71186061-openshift-default-5698544949-klmg5"] Oct 31 03:48:53.584715 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:53.584678 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:54.318198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:54.318159 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" path="/var/lib/kubelet/pods/0761e73a-f63e-4edc-958a-b96db8614d28/volumes" Oct 31 03:48:54.460679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:54.460647 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:56.502866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.502822 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:56.680081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.680043 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:56.912261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912223 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7"] Oct 31 03:48:56.912820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912806 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" Oct 31 03:48:56.912867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912822 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" Oct 31 03:48:56.912867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912841 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" Oct 31 03:48:56.912867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912846 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" Oct 31 03:48:56.913012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912935 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" Oct 31 03:48:56.913012 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.912946 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0761e73a-f63e-4edc-958a-b96db8614d28" containerName="istio-proxy" Oct 31 03:48:56.915346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.915331 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:56.969907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:56.969856 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bc91a3-dockercfg-tfv4d\"" Oct 31 03:48:57.081472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.081442 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bc91a3\"" Oct 31 03:48:57.102112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.098262 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7"] Oct 31 03:48:57.107038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.107000 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.107424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.107402 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qcj\" (UniqueName: \"kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj\") pod \"payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.109345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.109319 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js"] Oct 31 03:48:57.113235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.113212 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.132059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.132029 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js"] Oct 31 03:48:57.209251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.209143 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qcj\" (UniqueName: \"kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj\") pod \"payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.209251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.209195 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bc91a3-5889cc7877-854js\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.209535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.209506 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8z5\" (UniqueName: \"kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5\") pod \"payload-processing-e2e-api-a-bc91a3-5889cc7877-854js\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.209626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.209609 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.210496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.210468 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.271047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.271011 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qcj\" (UniqueName: \"kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj\") pod \"payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.311015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.310975 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8z5\" (UniqueName: \"kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5\") pod \"payload-processing-e2e-api-a-bc91a3-5889cc7877-854js\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.311429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.311402 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bc91a3-5889cc7877-854js\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.312284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.312261 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bc91a3-5889cc7877-854js\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.371992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.371964 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8z5\" (UniqueName: \"kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5\") pod \"payload-processing-e2e-api-a-bc91a3-5889cc7877-854js\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.437763 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.437727 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:57.530246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.530209 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:57.816343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.816313 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js"] Oct 31 03:48:57.817672 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:48:57.817647 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d9272b_bc18_4749_9818_cd6701024fd3.slice/crio-3fd67bdc65ca85063539fda5ac0018bb62c25550453d25a60d887503bac01dc0 WatchSource:0}: Error finding container 3fd67bdc65ca85063539fda5ac0018bb62c25550453d25a60d887503bac01dc0: Status 404 returned error can't find the container with id 3fd67bdc65ca85063539fda5ac0018bb62c25550453d25a60d887503bac01dc0 Oct 31 03:48:57.928979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:57.928949 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7"] Oct 31 03:48:57.930373 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:48:57.930346 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2273bfb2_857f_416f_82da_9550eb28a682.slice/crio-23ee54f4939a2d846c0221a318eb74bf34beafb7ab6ede3e662a968a460b5e58 WatchSource:0}: Error finding container 23ee54f4939a2d846c0221a318eb74bf34beafb7ab6ede3e662a968a460b5e58: Status 404 returned error can't find the container with id 23ee54f4939a2d846c0221a318eb74bf34beafb7ab6ede3e662a968a460b5e58 Oct 31 03:48:58.252519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.252484 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" event={"ID":"95d9272b-bc18-4749-9818-cd6701024fd3","Type":"ContainerStarted","Data":"4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649"} Oct 31 03:48:58.252704 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.252535 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" event={"ID":"95d9272b-bc18-4749-9818-cd6701024fd3","Type":"ContainerStarted","Data":"3fd67bdc65ca85063539fda5ac0018bb62c25550453d25a60d887503bac01dc0"} Oct 31 03:48:58.252793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.252701 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:48:58.254330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.254292 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" event={"ID":"2273bfb2-857f-416f-82da-9550eb28a682","Type":"ContainerStarted","Data":"a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0"} Oct 31 03:48:58.254471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.254337 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" event={"ID":"2273bfb2-857f-416f-82da-9550eb28a682","Type":"ContainerStarted","Data":"23ee54f4939a2d846c0221a318eb74bf34beafb7ab6ede3e662a968a460b5e58"} Oct 31 03:48:58.254471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.254386 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:48:58.373482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.373429 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" podStartSLOduration=1.373411829 podStartE2EDuration="1.373411829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:58.368886907 +0000 UTC m=+1766.165605177" watchObservedRunningTime="2026-10-31 03:48:58.373411829 +0000 UTC m=+1766.170130083" Oct 31 03:48:58.485786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.485725 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" podStartSLOduration=2.485704821 podStartE2EDuration="2.485704821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:58.483988035 +0000 UTC m=+1766.280706297" watchObservedRunningTime="2026-10-31 03:48:58.485704821 +0000 UTC m=+1766.282423079" Oct 31 03:48:58.866041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:48:58.866004 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:49:00.122114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:00.122045 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:02.568158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:02.565830 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:05.154155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.152297 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:05.561285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.561196 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn"] Oct 31 03:49:05.565131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.565068 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.656154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.656122 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:05.670772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.668332 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1180d818\"/\"openshift-service-ca.crt\"" Oct 31 03:49:05.672793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.672545 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1180d818\"/\"kube-root-ca.crt\"" Oct 31 03:49:05.698184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.698146 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1180d818\"/\"default-dockercfg-gb5fr\"" Oct 31 03:49:05.703170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.703112 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1180d818\"/\"e2e-lifecycle-model-1180d818-kserve-self-signed-certs\"" Oct 31 03:49:05.718129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.716278 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn"] Oct 31 03:49:05.744570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.743448 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59cw\" (UniqueName: \"kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.744570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.743622 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.744570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.743679 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.744570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.743724 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.744570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.743781 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.744570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.743850 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846195 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59cw\" (UniqueName: \"kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846299 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846341 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846382 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846430 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846585 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846495 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846791 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846835 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.846902 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.846868 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.848876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.848848 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.849398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.849378 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:05.908864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:05.908826 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59cw\" (UniqueName: \"kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw\") pod \"e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:06.181658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:06.181565 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:06.576594 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:49:06.576556 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod763a1185_9d80_411e_b963_6a16a9f19133.slice/crio-9e3d20cd1e06064c56e6bb6523105111a4b41afd5c9d05d78703073d05c84a08 WatchSource:0}: Error finding container 9e3d20cd1e06064c56e6bb6523105111a4b41afd5c9d05d78703073d05c84a08: Status 404 returned error can't find the container with id 9e3d20cd1e06064c56e6bb6523105111a4b41afd5c9d05d78703073d05c84a08 Oct 31 03:49:06.577190 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:06.577162 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn"] Oct 31 03:49:06.593626 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:49:06.593557 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:49:07.369988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:07.369943 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" event={"ID":"763a1185-9d80-411e-b963-6a16a9f19133","Type":"ContainerStarted","Data":"955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703"} Oct 31 03:49:07.370552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:07.370004 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" event={"ID":"763a1185-9d80-411e-b963-6a16a9f19133","Type":"ContainerStarted","Data":"9e3d20cd1e06064c56e6bb6523105111a4b41afd5c9d05d78703073d05c84a08"} Oct 31 03:49:07.490729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:07.490667 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" podStartSLOduration=2.490650053 podStartE2EDuration="2.490650053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:07.488791223 +0000 UTC m=+1775.285509506" watchObservedRunningTime="2026-10-31 03:49:07.490650053 +0000 UTC m=+1775.287368311" Oct 31 03:49:08.182499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:08.182455 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:08.440342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:08.440251 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:49:08.440342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:08.440317 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:09.266811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.266774 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:09.267413 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.267388 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:49:09.267552 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.267463 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:49:09.801300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.801252 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c"] Oct 31 03:49:09.841115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.834134 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.901469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.901435 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c"] Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.916778 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7c43a6-openshift-default-dockercfg-dpqgw\"" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930492 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930592 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930694 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930725 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930763 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930828 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.930896 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.931079 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.931157 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkq7q\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:09.942164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:09.932126 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.032910 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033184 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033503 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033611 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkq7q\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033688 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033755 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033843 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033871 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033913 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.033981 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.034322 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.034370 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.034955 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.035021 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.037604 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.060079 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.094880 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.095026 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkq7q\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q\") pod \"e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.099645 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.215617 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.323035 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.691179 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c"] Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:49:10.691610 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228357ef_0268_409f_bd42_73f83b0567de.slice/crio-67d50f6ab0c3a2141af7a58ffba6a05d8cc7abfc9b833afeae4ad618829280db WatchSource:0}: Error finding container 67d50f6ab0c3a2141af7a58ffba6a05d8cc7abfc9b833afeae4ad618829280db: Status 404 returned error can't find the container with id 67d50f6ab0c3a2141af7a58ffba6a05d8cc7abfc9b833afeae4ad618829280db Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.694584 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.694654 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:10.932300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:10.694685 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.408381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.408343 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" event={"ID":"228357ef-0268-409f-bd42-73f83b0567de","Type":"ContainerStarted","Data":"97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6"} Oct 31 03:49:11.408660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.408407 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" event={"ID":"228357ef-0268-409f-bd42-73f83b0567de","Type":"ContainerStarted","Data":"67d50f6ab0c3a2141af7a58ffba6a05d8cc7abfc9b833afeae4ad618829280db"} Oct 31 03:49:11.410273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.410247 4488 generic.go:356] "Generic (PLEG): container finished" podID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerID="e67379cc8282000c865d0b26b4c30314a0e7d833ff835595be5cdc1d6d8102fe" exitCode=0 Oct 31 03:49:11.410411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.410319 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" event={"ID":"0c937467-ba00-456f-a0cc-ef9f919855eb","Type":"ContainerDied","Data":"e67379cc8282000c865d0b26b4c30314a0e7d833ff835595be5cdc1d6d8102fe"} Oct 31 03:49:11.410761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.410740 4488 scope.go:122] "RemoveContainer" containerID="e67379cc8282000c865d0b26b4c30314a0e7d833ff835595be5cdc1d6d8102fe" Oct 31 03:49:11.412232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.412211 4488 generic.go:356] "Generic (PLEG): container finished" podID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerID="661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1" exitCode=0 Oct 31 03:49:11.412326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.412262 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" event={"ID":"4d7b08bd-1c06-4308-84ed-f71ef4043463","Type":"ContainerDied","Data":"661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1"} Oct 31 03:49:11.412571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.412544 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.412610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.412584 4488 scope.go:122] "RemoveContainer" containerID="661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1" Oct 31 03:49:11.412648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.412615 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.412681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.412651 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.414145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.414118 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.414241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.414181 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.414241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.414214 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:11.789847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:11.789743 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podStartSLOduration=2.789726737 podStartE2EDuration="2.789726737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:11.621827976 +0000 UTC m=+1779.418546233" watchObservedRunningTime="2026-10-31 03:49:11.789726737 +0000 UTC m=+1779.586444995" Oct 31 03:49:12.217536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.217504 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:12.237982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.235975 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:12.471019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.470925 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" event={"ID":"0c937467-ba00-456f-a0cc-ef9f919855eb","Type":"ContainerStarted","Data":"c967acdb07235b51de7482b990a342aeffd0f9c153c46f650f4073dab5b4b4c9"} Oct 31 03:49:12.478691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.478609 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" event={"ID":"4d7b08bd-1c06-4308-84ed-f71ef4043463","Type":"ContainerStarted","Data":"130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb"} Oct 31 03:49:12.479501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.479480 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:12.480377 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.480361 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:49:12.689166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.689137 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:49:12.689485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.689459 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:49:12.689581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:12.689509 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 03:49:13.448136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:13.447929 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:49:13.448688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:13.448161 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:49:13.448688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:13.448210 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:49:13.689539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:13.689505 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:49:13.689717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:13.689557 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 03:49:14.447115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:14.446476 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:49:14.447115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:14.446543 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:49:14.688993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:14.688962 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:49:14.689436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:14.689009 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 03:49:15.383293 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.383258 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:15.447118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.444544 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:49:15.447118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.444608 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:49:15.486115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.480223 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:49:15.581404 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.581374 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:15.689676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.689594 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:49:15.689676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.689653 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 03:49:15.786501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:15.786474 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:16.181778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.181747 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:16.194283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.194257 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:16.444811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.444511 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:49:16.444811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.444600 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:49:16.568228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.568193 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:49:16.688985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.688951 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:49:16.689298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:16.689270 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 03:49:17.445581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:17.445542 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 31 03:49:17.446264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:17.446230 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 31 03:49:17.689643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:17.689610 4488 patch_prober.go:29] interesting pod/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:49:17.689892 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:17.689673 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 03:49:18.449152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:18.449107 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:49:18.450331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:18.450310 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:49:18.694320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:18.694293 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:49:18.694606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:18.694581 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:49:18.695456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:18.695437 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:49:18.823403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:18.823313 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:19.062738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.062709 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:19.258114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.251877 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55"] Oct 31 03:49:19.258114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.256450 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.295131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.295057 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55"] Oct 31 03:49:19.314344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.314304 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7c43a6\"" Oct 31 03:49:19.354036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.354000 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.354299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.354070 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctd5\" (UniqueName: \"kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5\") pod \"payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.435417 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.435378 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4"] Oct 31 03:49:19.439339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.439315 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.466462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.466388 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.467548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.467489 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctd5\" (UniqueName: \"kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5\") pod \"payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.468549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.468520 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.474604 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.474574 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4"] Oct 31 03:49:19.507035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.507003 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7c43a6-dockercfg-v78kh\"" Oct 31 03:49:19.541387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.541319 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctd5\" (UniqueName: \"kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5\") pod \"payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.570583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.570510 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.570812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.570748 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zbn\" (UniqueName: \"kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn\") pod \"payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.576409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.576376 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:19.671995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.671955 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zbn\" (UniqueName: \"kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn\") pod \"payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.687117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.685947 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.689521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.687413 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.749200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.749151 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zbn\" (UniqueName: \"kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn\") pod \"payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.760583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.760550 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:19.771168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:19.771131 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:20.055272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.055239 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55"] Oct 31 03:49:20.056690 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:49:20.056665 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb32f56_d27d_4e57_a006_306de73598f8.slice/crio-bc30280b7407662e3d2876a706d7ffbad931e81ab131f38c88834050d3398006 WatchSource:0}: Error finding container bc30280b7407662e3d2876a706d7ffbad931e81ab131f38c88834050d3398006: Status 404 returned error can't find the container with id bc30280b7407662e3d2876a706d7ffbad931e81ab131f38c88834050d3398006 Oct 31 03:49:20.157469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.157445 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4"] Oct 31 03:49:20.158139 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:49:20.158109 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db94ead_53db_45ff_9735_a53dd2a0c38b.slice/crio-3063f3862143bb1998c4ea7428677826637ebf3acd4887b0bdbf95327b6a5797 WatchSource:0}: Error finding container 3063f3862143bb1998c4ea7428677826637ebf3acd4887b0bdbf95327b6a5797: Status 404 returned error can't find the container with id 3063f3862143bb1998c4ea7428677826637ebf3acd4887b0bdbf95327b6a5797 Oct 31 03:49:20.255668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.255636 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29"] Oct 31 03:49:20.258672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.258655 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.295375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.295343 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfz6\" (UniqueName: \"kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.295502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.295394 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.295560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.295543 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.335786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.335747 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29"] Oct 31 03:49:20.397583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.396932 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfz6\" (UniqueName: \"kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.397583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.397020 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.397583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.397264 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.397974 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.397950 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.405674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.405649 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:49:20.405942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.405923 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:49:20.410299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.410283 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.461274 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.461247 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfz6\" (UniqueName: \"kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6\") pod \"maas-api-revoke-keys-e2e-worker-w4-096812-qnj29\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.571774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.571744 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:20.609125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.607819 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" event={"ID":"3db94ead-53db-45ff-9735-a53dd2a0c38b","Type":"ContainerStarted","Data":"a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e"} Oct 31 03:49:20.609125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.607882 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" event={"ID":"3db94ead-53db-45ff-9735-a53dd2a0c38b","Type":"ContainerStarted","Data":"3063f3862143bb1998c4ea7428677826637ebf3acd4887b0bdbf95327b6a5797"} Oct 31 03:49:20.609125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.608028 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:20.609980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.609954 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" event={"ID":"bfb32f56-d27d-4e57-a006-306de73598f8","Type":"ContainerStarted","Data":"81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f"} Oct 31 03:49:20.610082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.610001 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" event={"ID":"bfb32f56-d27d-4e57-a006-306de73598f8","Type":"ContainerStarted","Data":"bc30280b7407662e3d2876a706d7ffbad931e81ab131f38c88834050d3398006"} Oct 31 03:49:20.610082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.610026 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:20.732492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.732437 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" podStartSLOduration=1.732420749 podStartE2EDuration="1.732420749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:20.730757575 +0000 UTC m=+1788.527475831" watchObservedRunningTime="2026-10-31 03:49:20.732420749 +0000 UTC m=+1788.529139049" Oct 31 03:49:20.879134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.878973 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" podStartSLOduration=1.878952322 podStartE2EDuration="1.878952322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:20.878332345 +0000 UTC m=+1788.675050604" watchObservedRunningTime="2026-10-31 03:49:20.878952322 +0000 UTC m=+1788.675670581" Oct 31 03:49:20.996906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:20.996871 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29"] Oct 31 03:49:20.996906 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:49:20.996891 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d46759_a515_4c20_af0e_06a15d3a454e.slice/crio-f0331a9030d73408a481a5f0121afbbdaccbff33aff6251ee9fea2be7cd7276d WatchSource:0}: Error finding container f0331a9030d73408a481a5f0121afbbdaccbff33aff6251ee9fea2be7cd7276d: Status 404 returned error can't find the container with id f0331a9030d73408a481a5f0121afbbdaccbff33aff6251ee9fea2be7cd7276d Oct 31 03:49:21.167117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:21.167011 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:21.621139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:21.621104 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" event={"ID":"95d46759-a515-4c20-af0e-06a15d3a454e","Type":"ContainerStarted","Data":"d6560d89dd3addac59d5a02f1a0582b386d985cf304c660ca13ba065c290d097"} Oct 31 03:49:21.621575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:21.621178 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" event={"ID":"95d46759-a515-4c20-af0e-06a15d3a454e","Type":"ContainerStarted","Data":"f0331a9030d73408a481a5f0121afbbdaccbff33aff6251ee9fea2be7cd7276d"} Oct 31 03:49:21.751656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:21.751598 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" podStartSLOduration=1.7515813439999999 podStartE2EDuration="1.751581344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:21.748922801 +0000 UTC m=+1789.545641060" watchObservedRunningTime="2026-10-31 03:49:21.751581344 +0000 UTC m=+1789.548299655" Oct 31 03:49:22.630660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:22.630625 4488 generic.go:356] "Generic (PLEG): container finished" podID="95d46759-a515-4c20-af0e-06a15d3a454e" containerID="d6560d89dd3addac59d5a02f1a0582b386d985cf304c660ca13ba065c290d097" exitCode=0 Oct 31 03:49:22.631162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:22.630684 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" event={"ID":"95d46759-a515-4c20-af0e-06a15d3a454e","Type":"ContainerDied","Data":"d6560d89dd3addac59d5a02f1a0582b386d985cf304c660ca13ba065c290d097"} Oct 31 03:49:23.811120 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.811062 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:23.850395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.850357 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca\") pod \"95d46759-a515-4c20-af0e-06a15d3a454e\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " Oct 31 03:49:23.850592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.850564 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca\") pod \"95d46759-a515-4c20-af0e-06a15d3a454e\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " Oct 31 03:49:23.850653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.850637 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6\" (UniqueName: \"kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6\") pod \"95d46759-a515-4c20-af0e-06a15d3a454e\" (UID: \"95d46759-a515-4c20-af0e-06a15d3a454e\") " Oct 31 03:49:23.850771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.850710 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca" pod "95d46759-a515-4c20-af0e-06a15d3a454e" (UID: "95d46759-a515-4c20-af0e-06a15d3a454e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:49:23.851272 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.851251 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:23.852821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.852798 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca" pod "95d46759-a515-4c20-af0e-06a15d3a454e" (UID: "95d46759-a515-4c20-af0e-06a15d3a454e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:49:23.852885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.852866 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6" pod "95d46759-a515-4c20-af0e-06a15d3a454e" (UID: "95d46759-a515-4c20-af0e-06a15d3a454e"). InnerVolumeSpecName "kube-api-access-9qfz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:23.953479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.953390 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d46759-a515-4c20-af0e-06a15d3a454e-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:23.953479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:23.953429 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfz6\" (UniqueName: \"kubernetes.io/projected/95d46759-a515-4c20-af0e-06a15d3a454e-kube-api-access-9qfz6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:24.647966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:24.647916 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" Oct 31 03:49:24.648188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:24.647965 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29" event={"ID":"95d46759-a515-4c20-af0e-06a15d3a454e","Type":"ContainerDied","Data":"f0331a9030d73408a481a5f0121afbbdaccbff33aff6251ee9fea2be7cd7276d"} Oct 31 03:49:24.648188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:24.648021 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0331a9030d73408a481a5f0121afbbdaccbff33aff6251ee9fea2be7cd7276d" Oct 31 03:49:24.773252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:24.773220 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs"] Oct 31 03:49:24.869488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:24.869454 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:26.519439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:26.519406 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:27.919045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:27.919017 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:31.625459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:31.625422 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:49:31.625930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:31.625497 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:49:31.629164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:31.629113 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:32.165536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:32.165503 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:32.323873 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:32.323838 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:34.774987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:34.774951 4488 scope.go:122] "RemoveContainer" containerID="1fba6c2b04016b6de0c9a0d1c88a9d1aedfa1fa480336b79c22ad28ee549dbef" Oct 31 03:49:34.786510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:34.786461 4488 scope.go:122] "RemoveContainer" containerID="82445b6b271ab14abbbd5a3d2c2b51a544de301d17e713a4c4014a4c3a4038ee" Oct 31 03:49:36.593722 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:49:36.593683 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:49:36.694265 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:36.694223 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:38.855888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:38.855851 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:49:39.814661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:39.814614 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" containerID="cri-o://be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8" gracePeriod=45 Oct 31 03:49:40.151927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.151886 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:49:40.237084 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237040 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir\") pod \"f22472fe-b5bb-4545-b3f4-1b589c55d349\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " Oct 31 03:49:40.237310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237152 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home\") pod \"f22472fe-b5bb-4545-b3f4-1b589c55d349\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " Oct 31 03:49:40.237310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237200 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb\" (UniqueName: \"kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb\") pod \"f22472fe-b5bb-4545-b3f4-1b589c55d349\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " Oct 31 03:49:40.237429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237342 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache\") pod \"f22472fe-b5bb-4545-b3f4-1b589c55d349\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " Oct 31 03:49:40.237429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237394 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm\") pod \"f22472fe-b5bb-4545-b3f4-1b589c55d349\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " Oct 31 03:49:40.237429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237381 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir" pod "f22472fe-b5bb-4545-b3f4-1b589c55d349" (UID: "f22472fe-b5bb-4545-b3f4-1b589c55d349"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.237600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237454 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home" pod "f22472fe-b5bb-4545-b3f4-1b589c55d349" (UID: "f22472fe-b5bb-4545-b3f4-1b589c55d349"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.237600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237476 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs\" (UniqueName: \"kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs\") pod \"f22472fe-b5bb-4545-b3f4-1b589c55d349\" (UID: \"f22472fe-b5bb-4545-b3f4-1b589c55d349\") " Oct 31 03:49:40.237600 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237572 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache" pod "f22472fe-b5bb-4545-b3f4-1b589c55d349" (UID: "f22472fe-b5bb-4545-b3f4-1b589c55d349"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.237963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237943 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.237963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237966 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.237963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.237978 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.239707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.239683 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs" pod "f22472fe-b5bb-4545-b3f4-1b589c55d349" (UID: "f22472fe-b5bb-4545-b3f4-1b589c55d349"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:49:40.239707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.239688 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm" pod "f22472fe-b5bb-4545-b3f4-1b589c55d349" (UID: "f22472fe-b5bb-4545-b3f4-1b589c55d349"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:40.239835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.239707 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb" pod "f22472fe-b5bb-4545-b3f4-1b589c55d349" (UID: "f22472fe-b5bb-4545-b3f4-1b589c55d349"). InnerVolumeSpecName "kube-api-access-ktrfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:40.341848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.341811 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22472fe-b5bb-4545-b3f4-1b589c55d349-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.341848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.341853 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktrfb\" (UniqueName: \"kubernetes.io/projected/f22472fe-b5bb-4545-b3f4-1b589c55d349-kube-api-access-ktrfb\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.342083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.341868 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22472fe-b5bb-4545-b3f4-1b589c55d349-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:49:40.824108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.824051 4488 generic.go:356] "Generic (PLEG): container finished" podID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerID="be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8" exitCode=0 Oct 31 03:49:40.824323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.824129 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" event={"ID":"f22472fe-b5bb-4545-b3f4-1b589c55d349","Type":"ContainerDied","Data":"be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8"} Oct 31 03:49:40.824323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.824147 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" Oct 31 03:49:40.824323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.824181 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs" event={"ID":"f22472fe-b5bb-4545-b3f4-1b589c55d349","Type":"ContainerDied","Data":"959dc14b3ac9ac8f919d70a11d4383f47ed5adb02e65c9ccfec5bc7f00b29321"} Oct 31 03:49:40.824323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.824208 4488 scope.go:122] "RemoveContainer" containerID="be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8" Oct 31 03:49:40.845779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.844641 4488 scope.go:122] "RemoveContainer" containerID="be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8" Oct 31 03:49:40.845779 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:49:40.845278 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8\": container with ID starting with be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8 not found: ID does not exist" containerID="be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8" Oct 31 03:49:40.845779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.845314 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8"} err="failed to get container status \"be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8\": rpc error: code = NotFound desc = could not find container \"be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8\": container with ID starting with be27f375231c28285016096aff91b997cc523d3f7e78175cc07dc4f003958ea8 not found: ID does not exist" Oct 31 03:49:40.999046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:40.999006 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs"] Oct 31 03:49:41.012168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:41.012133 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-096812/distinct-w4-096812-kserve-8c4d8f7b7-dd8hs"] Oct 31 03:49:41.802925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:41.802884 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:42.319649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:42.319605 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" path="/var/lib/kubelet/pods/f22472fe-b5bb-4545-b3f4-1b589c55d349/volumes" Oct 31 03:49:46.625051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:46.624977 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:50.122357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:50.122328 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:50.295638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:50.295590 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:51.463278 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:51.463240 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:52.006531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.006497 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:52.387547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.387506 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:52.542113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.541355 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl"] Oct 31 03:49:52.560388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.560357 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" Oct 31 03:49:52.560669 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.560647 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" Oct 31 03:49:52.560903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.560889 4488 cpu_manager.go:414] "containerMap: removing container" podUID="95d46759-a515-4c20-af0e-06a15d3a454e" containerName="revoke-keys" Oct 31 03:49:52.589212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.589166 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d46759-a515-4c20-af0e-06a15d3a454e" containerName="revoke-keys" Oct 31 03:49:52.589926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.589892 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d46759-a515-4c20-af0e-06a15d3a454e" containerName="revoke-keys" Oct 31 03:49:52.590160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.590137 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22472fe-b5bb-4545-b3f4-1b589c55d349" containerName="main" Oct 31 03:49:52.626971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.625545 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:52.670179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.664174 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl"] Oct 31 03:49:52.670179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.664337 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.720064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.720036 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-1783b3-openshift-default-dockercfg-2fskz\"" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.751225 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.751329 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpmb\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752069 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752239 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752354 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752446 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752511 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752593 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.753210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.752638 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854305 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854369 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854456 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854525 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854575 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854651 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854713 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854907 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.854975 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.854909 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.855119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.855009 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpmb\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.855119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.855027 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.855237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.855119 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.855237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.855201 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.855445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.855424 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.856947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.856921 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.857430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.857405 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.917191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.917160 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpmb\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.917430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.917412 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:52.935926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:52.935823 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:53.001135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.001069 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:53.289863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.289777 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:53.454938 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.454900 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl"] Oct 31 03:49:53.491677 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:49:53.491638 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c5632c_d89d_4db0_a838_b5b40db09ee8.slice/crio-cf87752708120b7dcbd913d81b10bc12ca8589a076653ee6eac0f5ee66c9dda1 WatchSource:0}: Error finding container cf87752708120b7dcbd913d81b10bc12ca8589a076653ee6eac0f5ee66c9dda1: Status 404 returned error can't find the container with id cf87752708120b7dcbd913d81b10bc12ca8589a076653ee6eac0f5ee66c9dda1 Oct 31 03:49:53.494690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.494651 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:53.494817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.494740 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:53.494817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.494791 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:49:53.975158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.975123 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" event={"ID":"07c5632c-d89d-4db0-a838-b5b40db09ee8","Type":"ContainerStarted","Data":"215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e"} Oct 31 03:49:53.975587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:53.975185 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" event={"ID":"07c5632c-d89d-4db0-a838-b5b40db09ee8","Type":"ContainerStarted","Data":"cf87752708120b7dcbd913d81b10bc12ca8589a076653ee6eac0f5ee66c9dda1"} Oct 31 03:49:54.001728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:54.001698 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:54.008058 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:54.008027 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:54.132252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:54.132200 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podStartSLOduration=2.13218142 podStartE2EDuration="2.13218142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:54.129036413 +0000 UTC m=+1821.925754672" watchObservedRunningTime="2026-10-31 03:49:54.13218142 +0000 UTC m=+1821.928899678" Oct 31 03:49:54.982962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:54.982936 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:54.984201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:54.984184 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:49:57.127200 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:57.127166 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:57.335166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:57.335133 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:57.976291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:49:57.976254 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:00.188984 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:00.188948 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:00.868781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:00.868706 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:01.098719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:01.098687 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:01.310959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:01.310840 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:02.662778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.662734 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb"] Oct 31 03:50:02.667540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.667510 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74"] Oct 31 03:50:02.667726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.667634 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.672184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.672144 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.724208 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.724062 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:50:02.724545 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.724262 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:50:02.724545 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.724429 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb"] Oct 31 03:50:02.748426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.748378 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.748650 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.748603 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm56g\" (UniqueName: \"kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.748714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.748645 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.748816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.748791 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.748957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.748939 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.749011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.748978 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fdw8\" (UniqueName: \"kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.775477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.775439 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74"] Oct 31 03:50:02.850690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.850654 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.850690 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.850696 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fdw8\" (UniqueName: \"kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.850969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.850802 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.850969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.850880 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm56g\" (UniqueName: \"kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.850969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.850909 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.850969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.850960 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.851497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.851467 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.851755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.851727 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.853543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.853521 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.853634 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.853535 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.912471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.912438 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fdw8\" (UniqueName: \"kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8\") pod \"maas-api-revoke-sub-c80ae1e5-5ww74\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:02.912666 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.912485 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm56g\" (UniqueName: \"kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g\") pod \"maas-api-revoke-sub-5d2b4bfc-2l2fb\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.986277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.986195 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:02.992517 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:02.992493 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:03.145354 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.145320 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6"] Oct 31 03:50:03.149961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.149938 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.204173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.203766 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bc91a3-serving-cert\"" Oct 31 03:50:03.210425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.210393 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6"] Oct 31 03:50:03.260330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.260240 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.260517 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.260432 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.260571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.260537 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.260671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.260657 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbsr\" (UniqueName: \"kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.358548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.358493 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:03.362169 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.362139 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbsr\" (UniqueName: \"kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.362323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.362267 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.362388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.362331 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.362388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.362374 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.362535 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:50:03.362515 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-serving-cert: secret "maas-api-e2e-api-a-bc91a3-serving-cert" not found Oct 31 03:50:03.362606 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:50:03.362596 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls podName:6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9 nodeName:}" failed. No retries permitted until 2026-10-31 03:50:03.862577471 +0000 UTC m=+1831.659295712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls") pod "maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" (UID: "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9") : secret "maas-api-e2e-api-a-bc91a3-serving-cert" not found Oct 31 03:50:03.362891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.362860 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.364970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.364947 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.408163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.407579 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb"] Oct 31 03:50:03.410414 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:50:03.410385 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbdd5d5_1df0_4c59_8603_4c554162332f.slice/crio-d537bc1f1df34878dae0d8f04f24ab466cef98cd6cf7721abff4dee926b1e149 WatchSource:0}: Error finding container d537bc1f1df34878dae0d8f04f24ab466cef98cd6cf7721abff4dee926b1e149: Status 404 returned error can't find the container with id d537bc1f1df34878dae0d8f04f24ab466cef98cd6cf7721abff4dee926b1e149 Oct 31 03:50:03.425188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.425152 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbsr\" (UniqueName: \"kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.457814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.457771 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74"] Oct 31 03:50:03.459680 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:50:03.459648 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775baf45_b922_4333_94a6_a0c03c46d84c.slice/crio-ba6235481d159ebf12a1af2836a73de0a79e69d8769f513b3a20299dd4a31826 WatchSource:0}: Error finding container ba6235481d159ebf12a1af2836a73de0a79e69d8769f513b3a20299dd4a31826: Status 404 returned error can't find the container with id ba6235481d159ebf12a1af2836a73de0a79e69d8769f513b3a20299dd4a31826 Oct 31 03:50:03.513772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.513685 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:03.783639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.783563 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:03.869151 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.869072 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:03.872000 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:03.871970 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:04.067815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.067726 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:04.067815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.067785 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" event={"ID":"2bbdd5d5-1df0-4c59-8603-4c554162332f","Type":"ContainerStarted","Data":"24d5a7f1f8ff7787ca22821974931baedd2b506d2a0fef792970dd343a7424ac"} Oct 31 03:50:04.068033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.067840 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" event={"ID":"2bbdd5d5-1df0-4c59-8603-4c554162332f","Type":"ContainerStarted","Data":"d537bc1f1df34878dae0d8f04f24ab466cef98cd6cf7721abff4dee926b1e149"} Oct 31 03:50:04.069684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.069660 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" event={"ID":"775baf45-b922-4333-94a6-a0c03c46d84c","Type":"ContainerStarted","Data":"8f453e12ab9211d2817fbe474140527a96847fe288f1bbbb7c2019dbbc71a58b"} Oct 31 03:50:04.069817 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.069704 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" event={"ID":"775baf45-b922-4333-94a6-a0c03c46d84c","Type":"ContainerStarted","Data":"ba6235481d159ebf12a1af2836a73de0a79e69d8769f513b3a20299dd4a31826"} Oct 31 03:50:04.188124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.184014 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" podStartSLOduration=2.183994267 podStartE2EDuration="2.183994267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:04.182881431 +0000 UTC m=+1831.979599688" watchObservedRunningTime="2026-10-31 03:50:04.183994267 +0000 UTC m=+1831.980712526" Oct 31 03:50:04.303247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.303190 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" podStartSLOduration=2.303174546 podStartE2EDuration="2.303174546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:04.299159328 +0000 UTC m=+1832.095877590" watchObservedRunningTime="2026-10-31 03:50:04.303174546 +0000 UTC m=+1832.099892805" Oct 31 03:50:04.452898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.452868 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6"] Oct 31 03:50:04.453774 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:50:04.453751 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1cf4d8_a4fe_4acc_a225_aaa71aece6d9.slice/crio-143d1cbeb07e36a2ab3b197d387a0f7165bd836e07ad80047fa455f6dfaddde9 WatchSource:0}: Error finding container 143d1cbeb07e36a2ab3b197d387a0f7165bd836e07ad80047fa455f6dfaddde9: Status 404 returned error can't find the container with id 143d1cbeb07e36a2ab3b197d387a0f7165bd836e07ad80047fa455f6dfaddde9 Oct 31 03:50:04.455542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:04.455520 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:50:05.079057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:05.079024 4488 generic.go:356] "Generic (PLEG): container finished" podID="2bbdd5d5-1df0-4c59-8603-4c554162332f" containerID="24d5a7f1f8ff7787ca22821974931baedd2b506d2a0fef792970dd343a7424ac" exitCode=0 Oct 31 03:50:05.079572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:05.079117 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" event={"ID":"2bbdd5d5-1df0-4c59-8603-4c554162332f","Type":"ContainerDied","Data":"24d5a7f1f8ff7787ca22821974931baedd2b506d2a0fef792970dd343a7424ac"} Oct 31 03:50:05.080617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:05.080595 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" event={"ID":"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9","Type":"ContainerStarted","Data":"143d1cbeb07e36a2ab3b197d387a0f7165bd836e07ad80047fa455f6dfaddde9"} Oct 31 03:50:05.082273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:05.082245 4488 generic.go:356] "Generic (PLEG): container finished" podID="775baf45-b922-4333-94a6-a0c03c46d84c" containerID="8f453e12ab9211d2817fbe474140527a96847fe288f1bbbb7c2019dbbc71a58b" exitCode=0 Oct 31 03:50:05.082387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:05.082293 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" event={"ID":"775baf45-b922-4333-94a6-a0c03c46d84c","Type":"ContainerDied","Data":"8f453e12ab9211d2817fbe474140527a96847fe288f1bbbb7c2019dbbc71a58b"} Oct 31 03:50:06.107261 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.107217 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" event={"ID":"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9","Type":"ContainerStarted","Data":"e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6"} Oct 31 03:50:06.107734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.107619 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:06.231236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.231175 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" podStartSLOduration=2.365336533 podStartE2EDuration="3.231159245s" totalImagesPullingTime="865.822712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:06.228782233 +0000 UTC m=+1834.025500493" watchObservedRunningTime="2026-10-31 03:50:06.231159245 +0000 UTC m=+1834.027877502" Oct 31 03:50:06.443809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.443782 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:06.447775 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.447751 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:06.479026 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.478994 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca\") pod \"775baf45-b922-4333-94a6-a0c03c46d84c\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " Oct 31 03:50:06.479236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.479035 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca\") pod \"2bbdd5d5-1df0-4c59-8603-4c554162332f\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " Oct 31 03:50:06.479236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.479119 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca\") pod \"2bbdd5d5-1df0-4c59-8603-4c554162332f\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " Oct 31 03:50:06.479236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.479210 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g\" (UniqueName: \"kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g\") pod \"2bbdd5d5-1df0-4c59-8603-4c554162332f\" (UID: \"2bbdd5d5-1df0-4c59-8603-4c554162332f\") " Oct 31 03:50:06.479429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.479253 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8\" (UniqueName: \"kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8\") pod \"775baf45-b922-4333-94a6-a0c03c46d84c\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " Oct 31 03:50:06.479429 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.479366 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca\") pod \"775baf45-b922-4333-94a6-a0c03c46d84c\" (UID: \"775baf45-b922-4333-94a6-a0c03c46d84c\") " Oct 31 03:50:06.479594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.479551 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca" pod "2bbdd5d5-1df0-4c59-8603-4c554162332f" (UID: "2bbdd5d5-1df0-4c59-8603-4c554162332f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:06.484305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.480053 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca" pod "775baf45-b922-4333-94a6-a0c03c46d84c" (UID: "775baf45-b922-4333-94a6-a0c03c46d84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:06.484305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.480370 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.484305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.480400 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.487142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.486404 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca" pod "775baf45-b922-4333-94a6-a0c03c46d84c" (UID: "775baf45-b922-4333-94a6-a0c03c46d84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:06.487142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.486652 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g" pod "2bbdd5d5-1df0-4c59-8603-4c554162332f" (UID: "2bbdd5d5-1df0-4c59-8603-4c554162332f"). InnerVolumeSpecName "kube-api-access-gm56g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.489152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.488532 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8" pod "775baf45-b922-4333-94a6-a0c03c46d84c" (UID: "775baf45-b922-4333-94a6-a0c03c46d84c"). InnerVolumeSpecName "kube-api-access-7fdw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:06.509614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.509573 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca" pod "2bbdd5d5-1df0-4c59-8603-4c554162332f" (UID: "2bbdd5d5-1df0-4c59-8603-4c554162332f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:06.583111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.581874 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775baf45-b922-4333-94a6-a0c03c46d84c-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.583111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.581911 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbdd5d5-1df0-4c59-8603-4c554162332f-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.583111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.581926 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm56g\" (UniqueName: \"kubernetes.io/projected/2bbdd5d5-1df0-4c59-8603-4c554162332f-kube-api-access-gm56g\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.583111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:06.581942 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fdw8\" (UniqueName: \"kubernetes.io/projected/775baf45-b922-4333-94a6-a0c03c46d84c-kube-api-access-7fdw8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:06.597616 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:50:06.597577 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:50:07.119359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:07.119312 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" event={"ID":"775baf45-b922-4333-94a6-a0c03c46d84c","Type":"ContainerDied","Data":"ba6235481d159ebf12a1af2836a73de0a79e69d8769f513b3a20299dd4a31826"} Oct 31 03:50:07.119861 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:07.119440 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74" Oct 31 03:50:07.119861 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:07.119758 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6235481d159ebf12a1af2836a73de0a79e69d8769f513b3a20299dd4a31826" Oct 31 03:50:07.122325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:07.122304 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" Oct 31 03:50:07.126322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:07.126281 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb" event={"ID":"2bbdd5d5-1df0-4c59-8603-4c554162332f","Type":"ContainerDied","Data":"d537bc1f1df34878dae0d8f04f24ab466cef98cd6cf7721abff4dee926b1e149"} Oct 31 03:50:07.126488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:07.126351 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d537bc1f1df34878dae0d8f04f24ab466cef98cd6cf7721abff4dee926b1e149" Oct 31 03:50:08.379370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:08.379312 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:08.668987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:08.668889 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:09.265385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:09.265283 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:09.476221 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:09.476183 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:12.127757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:12.127729 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:50:13.225511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:13.225479 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:13.381605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:13.381557 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:13.536032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:13.535919 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:16.526759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:16.526676 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:16.709587 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:16.709516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:17.298722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.298679 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:17.628238 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628206 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2"] Oct 31 03:50:17.628835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628821 4488 cpu_manager.go:414] "containerMap: removing container" podUID="2bbdd5d5-1df0-4c59-8603-4c554162332f" containerName="revoke-keys" Oct 31 03:50:17.628881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628836 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbdd5d5-1df0-4c59-8603-4c554162332f" containerName="revoke-keys" Oct 31 03:50:17.628881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628854 4488 cpu_manager.go:414] "containerMap: removing container" podUID="775baf45-b922-4333-94a6-a0c03c46d84c" containerName="revoke-keys" Oct 31 03:50:17.628881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628859 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775baf45-b922-4333-94a6-a0c03c46d84c" containerName="revoke-keys" Oct 31 03:50:17.628993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628983 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbdd5d5-1df0-4c59-8603-4c554162332f" containerName="revoke-keys" Oct 31 03:50:17.629032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.628996 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="775baf45-b922-4333-94a6-a0c03c46d84c" containerName="revoke-keys" Oct 31 03:50:17.634411 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.634393 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.691731 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.691696 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2"] Oct 31 03:50:17.736355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.736323 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:50:17.736355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.736363 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:50:17.737045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.737021 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthrx\" (UniqueName: \"kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.737277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.737258 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.737342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.737297 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.838865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.838817 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.838865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.838871 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.839186 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.839102 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthrx\" (UniqueName: \"kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.839553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.839529 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.841698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.841678 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.900782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.900700 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthrx\" (UniqueName: \"kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:17.949890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:17.949850 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:18.334611 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:18.334576 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2"] Oct 31 03:50:18.335571 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:50:18.335549 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d2e90e_f8df_482d_9713_96fa8c4477ff.slice/crio-0805e760b9c60f5a6f1b703ee11773b03ef4a69cd52de884557a376b5fa9a0de WatchSource:0}: Error finding container 0805e760b9c60f5a6f1b703ee11773b03ef4a69cd52de884557a376b5fa9a0de: Status 404 returned error can't find the container with id 0805e760b9c60f5a6f1b703ee11773b03ef4a69cd52de884557a376b5fa9a0de Oct 31 03:50:18.690509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:18.690407 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:19.254616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:19.254582 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" event={"ID":"f4d2e90e-f8df-482d-9713-96fa8c4477ff","Type":"ContainerStarted","Data":"d3dec56b83b8907750984beb799d3a517c9a75d6b83d954b6b7b50948b5d42b7"} Oct 31 03:50:19.254812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:19.254634 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" event={"ID":"f4d2e90e-f8df-482d-9713-96fa8c4477ff","Type":"ContainerStarted","Data":"0805e760b9c60f5a6f1b703ee11773b03ef4a69cd52de884557a376b5fa9a0de"} Oct 31 03:50:19.372305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:19.372251 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" podStartSLOduration=2.372235368 podStartE2EDuration="2.372235368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:19.370447048 +0000 UTC m=+1847.167165302" watchObservedRunningTime="2026-10-31 03:50:19.372235368 +0000 UTC m=+1847.168953698" Oct 31 03:50:20.263425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:20.263388 4488 generic.go:356] "Generic (PLEG): container finished" podID="f4d2e90e-f8df-482d-9713-96fa8c4477ff" containerID="d3dec56b83b8907750984beb799d3a517c9a75d6b83d954b6b7b50948b5d42b7" exitCode=0 Oct 31 03:50:20.263894 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:20.263476 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" event={"ID":"f4d2e90e-f8df-482d-9713-96fa8c4477ff","Type":"ContainerDied","Data":"d3dec56b83b8907750984beb799d3a517c9a75d6b83d954b6b7b50948b5d42b7"} Oct 31 03:50:21.442241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.442218 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:21.595970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.595915 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca\") pod \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " Oct 31 03:50:21.596164 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.596146 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca\") pod \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " Oct 31 03:50:21.596250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.596234 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx\" (UniqueName: \"kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx\") pod \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\" (UID: \"f4d2e90e-f8df-482d-9713-96fa8c4477ff\") " Oct 31 03:50:21.596378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.596351 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca" pod "f4d2e90e-f8df-482d-9713-96fa8c4477ff" (UID: "f4d2e90e-f8df-482d-9713-96fa8c4477ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:21.596929 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.596900 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:21.598673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.598649 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx" pod "f4d2e90e-f8df-482d-9713-96fa8c4477ff" (UID: "f4d2e90e-f8df-482d-9713-96fa8c4477ff"). InnerVolumeSpecName "kube-api-access-xthrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:21.599032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.599005 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca" pod "f4d2e90e-f8df-482d-9713-96fa8c4477ff" (UID: "f4d2e90e-f8df-482d-9713-96fa8c4477ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:21.697990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.697946 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xthrx\" (UniqueName: \"kubernetes.io/projected/f4d2e90e-f8df-482d-9713-96fa8c4477ff-kube-api-access-xthrx\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:21.697990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:21.697987 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d2e90e-f8df-482d-9713-96fa8c4477ff-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:50:22.283240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:22.283210 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" Oct 31 03:50:22.283240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:22.283217 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2" event={"ID":"f4d2e90e-f8df-482d-9713-96fa8c4477ff","Type":"ContainerDied","Data":"0805e760b9c60f5a6f1b703ee11773b03ef4a69cd52de884557a376b5fa9a0de"} Oct 31 03:50:22.283536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:22.283314 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0805e760b9c60f5a6f1b703ee11773b03ef4a69cd52de884557a376b5fa9a0de" Oct 31 03:50:23.838161 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:23.838084 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:23.996345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:23.996305 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:24.155416 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:24.155307 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:27.476785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:27.476741 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:29.059105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:29.059058 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:29.214985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:29.214956 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:29.371694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:29.371665 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:33.989847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:33.989809 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:34.152547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:34.152511 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:34.315062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:34.314803 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:34.823446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:34.823411 4488 scope.go:122] "RemoveContainer" containerID="df92f25cc709a5940e20eaea7c070db0ec1f1abc4a61f704c688612a6f144837" Oct 31 03:50:34.833795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:34.833775 4488 scope.go:122] "RemoveContainer" containerID="6f43a2e6e6d10b0d7c0068704c6bd42e5aac3d80d1831008ddcf6b1041968ecd" Oct 31 03:50:34.843247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:34.843224 4488 scope.go:122] "RemoveContainer" containerID="f524c0611f07f2ac184e42470fe8d12894a8f29347a205d405216bbf1df8ef1d" Oct 31 03:50:36.590735 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:50:36.590691 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:50:38.811387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:38.811347 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:43.597456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:43.597422 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:46.800725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:46.800676 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:50:48.453168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:48.453071 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:52.796312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:52.796272 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:54.241377 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:54.241328 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:54.400192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:54.400141 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:54.574113 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:54.573991 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:54.814391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:54.814357 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:54.913701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:54.913664 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2"] Oct 31 03:50:54.960569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:54.960534 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e38e21-qjmx2"] Oct 31 03:50:55.344179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:55.344080 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:56.319842 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:56.319797 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d2e90e-f8df-482d-9713-96fa8c4477ff" path="/var/lib/kubelet/pods/f4d2e90e-f8df-482d-9713-96fa8c4477ff/volumes" Oct 31 03:50:58.521304 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:58.521255 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:50:58.521819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:58.521343 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:59.715487 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:50:59.715440 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:00.823871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.823781 4488 generic.go:356] "Generic (PLEG): container finished" podID="228357ef-0268-409f-bd42-73f83b0567de" containerID="97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6" exitCode=0 Oct 31 03:51:00.823871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.823857 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" event={"ID":"228357ef-0268-409f-bd42-73f83b0567de","Type":"ContainerDied","Data":"97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6"} Oct 31 03:51:00.824556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.824438 4488 scope.go:122] "RemoveContainer" containerID="97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6" Oct 31 03:51:00.826124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.826073 4488 generic.go:356] "Generic (PLEG): container finished" podID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerID="57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2" exitCode=0 Oct 31 03:51:00.826256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.826147 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" event={"ID":"8114c66e-0643-42d4-9ee0-e9c9d1923852","Type":"ContainerDied","Data":"57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2"} Oct 31 03:51:00.826496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.826464 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:00.826572 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.826556 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:00.826616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.826599 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:00.826712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.826697 4488 scope.go:122] "RemoveContainer" containerID="57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2" Oct 31 03:51:00.828355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.828327 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:00.828444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.828393 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:00.828444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:00.828430 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:01.839179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:01.839140 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" event={"ID":"8114c66e-0643-42d4-9ee0-e9c9d1923852","Type":"ContainerStarted","Data":"f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27"} Oct 31 03:51:01.841439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:01.841416 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" event={"ID":"228357ef-0268-409f-bd42-73f83b0567de","Type":"ContainerStarted","Data":"5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b"} Oct 31 03:51:02.220184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.217536 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:51:02.220184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.217684 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:51:02.220184 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.217730 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:51:02.425222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.425182 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:02.644115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.635962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:02.666054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.666025 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:51:02.667173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.667145 4488 patch_prober.go:29] interesting pod/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:51:02.667292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.667202 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:51:02.773924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.773395 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd"] Oct 31 03:51:02.773924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.773648 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" containerID="cri-o://20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907" gracePeriod=30 Oct 31 03:51:02.795233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:02.795145 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:03.248116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:03.242068 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:51:03.248116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:03.242159 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:51:03.672021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:03.670438 4488 patch_prober.go:29] interesting pod/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:51:03.672021 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:03.670506 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:51:04.224707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:04.224650 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:51:04.225054 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:04.225023 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:51:04.667140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:04.665800 4488 patch_prober.go:29] interesting pod/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:51:04.667140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:04.665859 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:51:04.771210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:04.771177 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:05.236326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:05.236284 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:51:05.236698 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:05.236665 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:51:05.499722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:05.499597 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:05.658432 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:05.658388 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:05.665860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:05.665386 4488 patch_prober.go:29] interesting pod/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:51:05.665860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:05.665474 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:51:06.197109 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:06.197063 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:51:06.219147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:06.216518 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:51:06.219147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:06.216583 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:51:06.600996 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:51:06.600945 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:51:06.666119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:06.665807 4488 patch_prober.go:29] interesting pod/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 03:51:06.666119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:06.665884 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 03:51:07.223115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:07.222525 4488 patch_prober.go:29] interesting pod/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 03:51:07.223115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:07.222607 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 03:51:07.745938 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:07.745892 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:51:07.748624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:07.748599 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:51:08.088748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.088716 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:51:08.184446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.184414 4488 generic.go:356] "Generic (PLEG): container finished" podID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerID="20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907" exitCode=0 Oct 31 03:51:08.184647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.184466 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" Oct 31 03:51:08.184647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.184492 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" event={"ID":"ccf94248-c7de-465f-9f12-f9a0275f5a5d","Type":"ContainerDied","Data":"20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907"} Oct 31 03:51:08.184647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.184546 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd" event={"ID":"ccf94248-c7de-465f-9f12-f9a0275f5a5d","Type":"ContainerDied","Data":"9926910c045b5c12373b716e172ae7c423ab301fdf768bc7d7687fd09a65de58"} Oct 31 03:51:08.184647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.184571 4488 scope.go:122] "RemoveContainer" containerID="20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907" Oct 31 03:51:08.186956 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.186937 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.186984 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187036 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187009 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187205 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187165 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187209 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187241 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187422 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187436 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:08.187588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187441 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:51:08.187588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187493 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187534 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs\") pod \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\" (UID: \"ccf94248-c7de-465f-9f12-f9a0275f5a5d\") " Oct 31 03:51:08.187914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187656 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:08.187914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.187756 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:08.188135 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.188069 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:08.188346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.188324 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.188346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.188354 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.188505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.188370 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.188505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.188384 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.189625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.189603 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "kube-api-access-7t4m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:08.189732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.189700 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:51:08.189851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.189834 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:08.190496 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.190474 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token" pod "ccf94248-c7de-465f-9f12-f9a0275f5a5d" (UID: "ccf94248-c7de-465f-9f12-f9a0275f5a5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:08.222605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.222571 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:51:08.222920 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.222878 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:51:08.223724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.223705 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:51:08.224494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.224475 4488 scope.go:122] "RemoveContainer" containerID="14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a" Oct 31 03:51:08.239866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.239838 4488 scope.go:122] "RemoveContainer" containerID="20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907" Oct 31 03:51:08.240419 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:51:08.240386 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907\": container with ID starting with 20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907 not found: ID does not exist" containerID="20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907" Oct 31 03:51:08.240529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.240428 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907"} err="failed to get container status \"20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907\": rpc error: code = NotFound desc = could not find container \"20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907\": container with ID starting with 20034c49aa56ba3ec94c826ba84b4c1322f8b3b50892260a752edc3e06e8b907 not found: ID does not exist" Oct 31 03:51:08.240529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.240457 4488 scope.go:122] "RemoveContainer" containerID="14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a" Oct 31 03:51:08.240913 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:51:08.240882 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a\": container with ID starting with 14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a not found: ID does not exist" containerID="14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a" Oct 31 03:51:08.240994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.240925 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a"} err="failed to get container status \"14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a\": rpc error: code = NotFound desc = could not find container \"14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a\": container with ID starting with 14afce14f78fb99292e93db332d1516a8aca3f0e810ccacca0a6df06197bbd5a not found: ID does not exist" Oct 31 03:51:08.290107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.290029 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.290107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.290079 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.290107 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.290109 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.290396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.290124 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4m9\" (UniqueName: \"kubernetes.io/projected/ccf94248-c7de-465f-9f12-f9a0275f5a5d-kube-api-access-7t4m9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.290396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.290137 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf94248-c7de-465f-9f12-f9a0275f5a5d-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:08.603193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.603101 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd"] Oct 31 03:51:08.610627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:08.610591 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e38e21-openshift-default-7b89644589-2vkvd"] Oct 31 03:51:09.614196 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:09.614151 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:09.774286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:09.774254 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:09.979080 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:09.979003 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:10.318799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:10.318715 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" path="/var/lib/kubelet/pods/ccf94248-c7de-465f-9f12-f9a0275f5a5d/volumes" Oct 31 03:51:14.550528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:14.550498 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:16.267579 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:16.267547 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:16.489886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:16.489839 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:19.421260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:19.421227 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:23.668115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:23.658347 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:25.220114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:25.220041 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:25.517246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:25.517159 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:25.671582 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:25.671553 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:25.786569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:25.786482 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29"] Oct 31 03:51:25.835982 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:25.835942 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-096812-qnj29"] Oct 31 03:51:26.210883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:26.210814 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:26.318688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:26.318654 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d46759-a515-4c20-af0e-06a15d3a454e" path="/var/lib/kubelet/pods/95d46759-a515-4c20-af0e-06a15d3a454e/volumes" Oct 31 03:51:26.679993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:26.679962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:27.389187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:27.389147 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:27.827142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:27.826783 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:28.065116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:28.039484 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:28.647643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:28.647606 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:30.775414 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:30.775379 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:30.938391 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:30.938358 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:31.093605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:31.093565 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:32.993456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:32.993418 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:33.185999 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:33.185938 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:33.350973 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:33.350943 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:33.722853 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:33.722732 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:33.884714 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:33.884680 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:34.054927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:34.054828 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2"] Oct 31 03:51:34.055436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:34.055163 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" containerID="cri-o://e8768d2d3c7991260e8ed0a01817013ea86b748ec65ae3de96983a1619edac69" gracePeriod=30 Oct 31 03:51:34.066367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:34.066336 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:34.068630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:34.068586 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46"] Oct 31 03:51:34.068950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:34.068906 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" containerID="cri-o://a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb" gracePeriod=30 Oct 31 03:51:36.217863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:36.217814 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:36.379112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:36.377908 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:36.581535 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:36.577017 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:36.684244 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:51:36.684210 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:51:39.411694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.411662 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:51:39.471395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471359 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.471674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471627 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.471872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471684 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.471872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471704 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.471872 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471790 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.472338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471886 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.472338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.471955 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.472338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.472071 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.472338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.472124 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.472338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.472298 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4\") pod \"17c44308-7237-4f0a-8011-6ca53cf7c381\" (UID: \"17c44308-7237-4f0a-8011-6ca53cf7c381\") " Oct 31 03:51:39.473138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.473024 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.473138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.473105 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.473270 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.473187 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.473331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.473269 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:51:39.473560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.473442 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.478477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.475950 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "kube-api-access-cqfv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:39.478477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.475947 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:39.478477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.477488 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:51:39.478477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.478443 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy" pod "17c44308-7237-4f0a-8011-6ca53cf7c381" (UID: "17c44308-7237-4f0a-8011-6ca53cf7c381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.574441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574407 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574437 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574447 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c44308-7237-4f0a-8011-6ca53cf7c381-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574455 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574464 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfv4\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-kube-api-access-cqfv4\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574474 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574484 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.574676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.574492 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c44308-7237-4f0a-8011-6ca53cf7c381-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.620594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.620507 4488 generic.go:356] "Generic (PLEG): container finished" podID="3c7949f7-535d-4204-aa14-28761776676f" containerID="e8768d2d3c7991260e8ed0a01817013ea86b748ec65ae3de96983a1619edac69" exitCode=0 Oct 31 03:51:39.620805 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.620587 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" event={"ID":"3c7949f7-535d-4204-aa14-28761776676f","Type":"ContainerDied","Data":"e8768d2d3c7991260e8ed0a01817013ea86b748ec65ae3de96983a1619edac69"} Oct 31 03:51:39.622812 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.622786 4488 generic.go:356] "Generic (PLEG): container finished" podID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerID="a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb" exitCode=0 Oct 31 03:51:39.622972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.622895 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" Oct 31 03:51:39.622972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.622950 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" event={"ID":"17c44308-7237-4f0a-8011-6ca53cf7c381","Type":"ContainerDied","Data":"a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb"} Oct 31 03:51:39.623105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.622995 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46" event={"ID":"17c44308-7237-4f0a-8011-6ca53cf7c381","Type":"ContainerDied","Data":"a881399a815c816e830cddacd57faf5f0520042cd6124c1e75ec69ef269fc812"} Oct 31 03:51:39.623105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.623024 4488 scope.go:122] "RemoveContainer" containerID="a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb" Oct 31 03:51:39.656591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.656456 4488 scope.go:122] "RemoveContainer" containerID="ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1" Oct 31 03:51:39.687484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.684256 4488 scope.go:122] "RemoveContainer" containerID="a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb" Oct 31 03:51:39.687484 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:51:39.685839 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb\": container with ID starting with a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb not found: ID does not exist" containerID="a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb" Oct 31 03:51:39.687484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.685877 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb"} err="failed to get container status \"a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb\": rpc error: code = NotFound desc = could not find container \"a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb\": container with ID starting with a0c07a63571f9f8e53b150a39585b85e75aa4c466d1c8fe0203c26954f11f6eb not found: ID does not exist" Oct 31 03:51:39.687484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.685902 4488 scope.go:122] "RemoveContainer" containerID="ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1" Oct 31 03:51:39.687484 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:51:39.686178 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1\": container with ID starting with ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1 not found: ID does not exist" containerID="ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1" Oct 31 03:51:39.687484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.686205 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1"} err="failed to get container status \"ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1\": rpc error: code = NotFound desc = could not find container \"ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1\": container with ID starting with ef9b5197d240826ab500fdb4eaf3218dab51ffb19e466d24eb50550f7fb269b1 not found: ID does not exist" Oct 31 03:51:39.793839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.793806 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46"] Oct 31 03:51:39.798804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.798772 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-kfp46"] Oct 31 03:51:39.832758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.832731 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:51:39.880019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.879919 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880005 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880139 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880174 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880219 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.880305 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880299 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880398 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880450 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880528 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880502 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880530 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.880702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880568 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:51:39.880808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880742 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket\") pod \"3c7949f7-535d-4204-aa14-28761776676f\" (UID: \"3c7949f7-535d-4204-aa14-28761776676f\") " Oct 31 03:51:39.880808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.880753 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.881104 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.881015 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.881739 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.881709 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.881859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.881742 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.881859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.881757 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7949f7-535d-4204-aa14-28761776676f-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.881859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.881772 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.881859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.881786 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.882686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.882665 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:39.882768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.882681 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:51:39.882768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.882753 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:39.883436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.883416 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx" pod "3c7949f7-535d-4204-aa14-28761776676f" (UID: "3c7949f7-535d-4204-aa14-28761776676f"). InnerVolumeSpecName "kube-api-access-pb7cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:39.983540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.983495 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.983540 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.983540 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7cx\" (UniqueName: \"kubernetes.io/projected/3c7949f7-535d-4204-aa14-28761776676f-kube-api-access-pb7cx\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.983792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.983556 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7949f7-535d-4204-aa14-28761776676f-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:39.983792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:39.983568 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7949f7-535d-4204-aa14-28761776676f-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:51:40.318228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:40.318142 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" path="/var/lib/kubelet/pods/17c44308-7237-4f0a-8011-6ca53cf7c381/volumes" Oct 31 03:51:40.634215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:40.634180 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" event={"ID":"3c7949f7-535d-4204-aa14-28761776676f","Type":"ContainerDied","Data":"922d0b8745b7d9d25a497596258c8a755ebf0ea89ebea573b2964ead424bc492"} Oct 31 03:51:40.634215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:40.634206 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2" Oct 31 03:51:40.634776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:40.634247 4488 scope.go:122] "RemoveContainer" containerID="e8768d2d3c7991260e8ed0a01817013ea86b748ec65ae3de96983a1619edac69" Oct 31 03:51:40.800374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:40.800290 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2"] Oct 31 03:51:41.170661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:40.850222 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-smtn2"] Oct 31 03:51:41.170661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.025262 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:41.170661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.025335 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:41.275779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.275741 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:41.643773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.643741 4488 generic.go:356] "Generic (PLEG): container finished" podID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerID="215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e" exitCode=0 Oct 31 03:51:41.644264 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.643811 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" event={"ID":"07c5632c-d89d-4db0-a838-b5b40db09ee8","Type":"ContainerDied","Data":"215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e"} Oct 31 03:51:41.644361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.644344 4488 scope.go:122] "RemoveContainer" containerID="215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e" Oct 31 03:51:41.646209 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.646180 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:41.646303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.646254 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:41.646303 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:41.646287 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:51:42.319462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:42.319432 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7949f7-535d-4204-aa14-28761776676f" path="/var/lib/kubelet/pods/3c7949f7-535d-4204-aa14-28761776676f/volumes" Oct 31 03:51:42.655737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:42.655697 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" event={"ID":"07c5632c-d89d-4db0-a838-b5b40db09ee8","Type":"ContainerStarted","Data":"65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f"} Oct 31 03:51:43.001955 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:43.001869 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:51:43.002237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:43.002211 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:51:43.002298 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:43.002269 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:51:44.002460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:44.002380 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:51:44.002460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:44.002446 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:51:45.002476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:45.002438 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:51:45.002887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:45.002514 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:51:46.002234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:46.002197 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:51:46.002408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:46.002269 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:51:46.229860 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:46.229831 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:47.002210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:47.002172 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:51:47.002409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:47.002246 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:51:48.001970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:48.001931 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:51:48.002456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:48.002019 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 03:51:49.006260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:49.006221 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:51:49.006749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:49.006734 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:51:49.007524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:49.007505 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:51:51.539351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:51.539311 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:56.761533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:56.761492 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:58.400580 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:58.400512 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:58.735477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:58.735379 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb"] Oct 31 03:51:58.993519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:58.993433 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:59.151560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:59.151522 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:59.801977 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:51:59.801927 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:00.186202 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:00.186039 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:02.005336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:02.005298 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:02.416346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:02.416305 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt"] Oct 31 03:52:02.418596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:02.418552 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" containerID="cri-o://4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22" gracePeriod=30 Oct 31 03:52:03.736707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:03.736659 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerName="payload-pre-processing" containerID="cri-o://77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a" gracePeriod=25 Oct 31 03:52:03.973602 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:03.973565 4488 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:52:03.973771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:03.973633 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:52:04.281560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:04.281525 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:04.589029 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:04.588989 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:04.745027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:04.744972 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:06.350319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:06.350282 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:52:06.591327 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:06.591292 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:52:07.172959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.172925 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:07.847182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.847140 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:52:07.914797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.914758 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.914837 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.914891 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915024 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915064 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915071 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:07.915224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915207 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915266 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:07.915454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915311 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915381 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915400 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:07.915454 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915434 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn\") pod \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\" (UID: \"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22\") " Oct 31 03:52:07.915708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915467 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:07.915708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.915569 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:07.916193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.916162 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:07.916193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.916188 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:07.916378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.916201 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:07.916378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.916214 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:07.916378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.916225 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:07.917830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.917803 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "kube-api-access-swpvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:07.917830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.917817 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:52:07.918280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.918255 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:07.918375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.918284 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token" pod "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" (UID: "9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:07.997019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.996923 4488 generic.go:356] "Generic (PLEG): container finished" podID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerID="4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22" exitCode=0 Oct 31 03:52:07.997019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.996976 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" Oct 31 03:52:07.997019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.997009 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" event={"ID":"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22","Type":"ContainerDied","Data":"4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22"} Oct 31 03:52:07.997329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.997071 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt" event={"ID":"9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22","Type":"ContainerDied","Data":"f371485d33220f88861cd5843fd2bc8f2a7f7399a702c3032ea24c7a0fffa458"} Oct 31 03:52:07.997329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:07.997116 4488 scope.go:122] "RemoveContainer" containerID="4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22" Oct 31 03:52:08.009248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.009225 4488 scope.go:122] "RemoveContainer" containerID="4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22" Oct 31 03:52:08.009590 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:08.009557 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22\": container with ID starting with 4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22 not found: ID does not exist" containerID="4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22" Oct 31 03:52:08.009685 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.009606 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22"} err="failed to get container status \"4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22\": rpc error: code = NotFound desc = could not find container \"4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22\": container with ID starting with 4f4d66a83e581f66c1435def0a99dd4e26171b96b5efe019649052154dba1d22 not found: ID does not exist" Oct 31 03:52:08.017570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.017539 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:08.017570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.017570 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:08.017734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.017581 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swpvn\" (UniqueName: \"kubernetes.io/projected/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-kube-api-access-swpvn\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:08.017734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.017590 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:08.120347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.120309 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:52:08.144220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.144182 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt"] Oct 31 03:52:08.147670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.147642 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-2zvbt"] Oct 31 03:52:08.318645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.318564 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" path="/var/lib/kubelet/pods/9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22/volumes" Oct 31 03:52:08.418557 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.418518 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:08.653247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:08.652769 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:10.517831 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:10.517797 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:10.676112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:10.675383 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:12.099533 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:12.099499 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:16.350594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:16.350557 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:52:16.857765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:16.857731 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:20.471491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:20.471451 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:20.670721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:20.670687 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:21.709961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:21.709905 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:23.850129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850064 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb"] Oct 31 03:52:23.850844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850819 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850847 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850869 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850876 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850889 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f4d2e90e-f8df-482d-9713-96fa8c4477ff" containerName="revoke-keys" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850896 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d2e90e-f8df-482d-9713-96fa8c4477ff" containerName="revoke-keys" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850924 4488 cpu_manager.go:414] "containerMap: removing container" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" Oct 31 03:52:23.850943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850932 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850952 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850960 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.850998 4488 cpu_manager.go:414] "containerMap: removing container" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851007 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851051 4488 cpu_manager.go:414] "containerMap: removing container" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851059 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851253 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851272 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7949f7-535d-4204-aa14-28761776676f" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851284 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c44308-7237-4f0a-8011-6ca53cf7c381" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851299 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbc629c-1eb1-423f-bd9d-37d7f2cb8c22" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851313 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" Oct 31 03:52:23.851328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851325 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf94248-c7de-465f-9f12-f9a0275f5a5d" containerName="istio-proxy" Oct 31 03:52:23.851885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.851335 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d2e90e-f8df-482d-9713-96fa8c4477ff" containerName="revoke-keys" Oct 31 03:52:23.853867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:23.853846 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.036944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.036909 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037055 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037102 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037191 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037217 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037370 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037489 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037474 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037562 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.037644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.037628 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bsc\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.043296 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.043277 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:52:24.083889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.083855 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb"] Oct 31 03:52:24.139314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.139271 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume\") pod \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " Oct 31 03:52:24.139583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.139566 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz\" (UniqueName: \"kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz\") pod \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\" (UID: \"89e2ddfc-6736-4776-b5f4-c3b7982106b5\") " Oct 31 03:52:24.139672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.139621 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume" pod "89e2ddfc-6736-4776-b5f4-c3b7982106b5" (UID: "89e2ddfc-6736-4776-b5f4-c3b7982106b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:24.139976 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.139936 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.139987 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140059 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140045 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140083 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140239 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140368 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140327 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140414 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140434 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140496 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bsc\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140539 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140604 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140642 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.140903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140871 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e2ddfc-6736-4776-b5f4-c3b7982106b5-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:24.141006 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.140976 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.141233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.141213 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.142064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.142034 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz" pod "89e2ddfc-6736-4776-b5f4-c3b7982106b5" (UID: "89e2ddfc-6736-4776-b5f4-c3b7982106b5"). InnerVolumeSpecName "kube-api-access-nxwjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:24.142630 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.142604 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.143117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.143075 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.223866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.223827 4488 generic.go:356] "Generic (PLEG): container finished" podID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerID="77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a" exitCode=0 Oct 31 03:52:24.224126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.223898 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" Oct 31 03:52:24.224126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.223925 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" event={"ID":"89e2ddfc-6736-4776-b5f4-c3b7982106b5","Type":"ContainerDied","Data":"77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a"} Oct 31 03:52:24.224126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.224011 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb" event={"ID":"89e2ddfc-6736-4776-b5f4-c3b7982106b5","Type":"ContainerDied","Data":"3dc2d4d339a15b71a937db62a1065952dc6b0c71bfaf32c14752db9f03fcfc9d"} Oct 31 03:52:24.224126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.224042 4488 scope.go:122] "RemoveContainer" containerID="77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a" Oct 31 03:52:24.235366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.235336 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.237319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.236962 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bsc\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.237319 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.237196 4488 scope.go:122] "RemoveContainer" containerID="77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a" Oct 31 03:52:24.237553 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:24.237528 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a\": container with ID starting with 77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a not found: ID does not exist" containerID="77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a" Oct 31 03:52:24.237616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.237563 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a"} err="failed to get container status \"77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a\": rpc error: code = NotFound desc = could not find container \"77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a\": container with ID starting with 77c055fdb2a25c8b1f1efbbd6e5fda87675a1bdc68266fdaa317b2d8a8f2121a not found: ID does not exist" Oct 31 03:52:24.242709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.242682 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwjz\" (UniqueName: \"kubernetes.io/projected/89e2ddfc-6736-4776-b5f4-c3b7982106b5-kube-api-access-nxwjz\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:24.394742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.394705 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb"] Oct 31 03:52:24.398614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.398585 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-dblmb"] Oct 31 03:52:24.467749 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.467698 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:24.903615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.903587 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb"] Oct 31 03:52:24.904189 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:52:24.904162 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c47b296_063b_4ae9_beb1_27f6764afcdd.slice/crio-90a5d77488e976840374bbc7e216b69877b860a4467694a647ed609919b39db7 WatchSource:0}: Error finding container 90a5d77488e976840374bbc7e216b69877b860a4467694a647ed609919b39db7: Status 404 returned error can't find the container with id 90a5d77488e976840374bbc7e216b69877b860a4467694a647ed609919b39db7 Oct 31 03:52:24.907571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.907529 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:52:24.907703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.907617 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:52:24.907703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:24.907692 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:52:25.235002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.234905 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" event={"ID":"0c47b296-063b-4ae9-beb1-27f6764afcdd","Type":"ContainerStarted","Data":"3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216"} Oct 31 03:52:25.235002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.234990 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" event={"ID":"0c47b296-063b-4ae9-beb1-27f6764afcdd","Type":"ContainerStarted","Data":"90a5d77488e976840374bbc7e216b69877b860a4467694a647ed609919b39db7"} Oct 31 03:52:25.358686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.358622 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podStartSLOduration=2.358602726 podStartE2EDuration="2.358602726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:25.355741373 +0000 UTC m=+1973.152459634" watchObservedRunningTime="2026-10-31 03:52:25.358602726 +0000 UTC m=+1973.155320985" Oct 31 03:52:25.468468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.468428 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:25.470406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.470376 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:52:25.470544 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.470432 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:52:25.843848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.843808 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw"] Oct 31 03:52:25.844108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.844042 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" podUID="f88f2d72-7161-447d-9b3e-0731e2448c48" containerName="maas-api" containerID="cri-o://d0b8f525d811152edf08e1f6bb2cbabe3f4b52e67538fd652a3195bcf56f9f4a" gracePeriod=30 Oct 31 03:52:25.949365 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:25.949324 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:26.387665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.387617 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" path="/var/lib/kubelet/pods/89e2ddfc-6736-4776-b5f4-c3b7982106b5/volumes" Oct 31 03:52:26.394044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.394010 4488 generic.go:356] "Generic (PLEG): container finished" podID="f88f2d72-7161-447d-9b3e-0731e2448c48" containerID="d0b8f525d811152edf08e1f6bb2cbabe3f4b52e67538fd652a3195bcf56f9f4a" exitCode=0 Oct 31 03:52:26.394283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.394077 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" event={"ID":"f88f2d72-7161-447d-9b3e-0731e2448c48","Type":"ContainerDied","Data":"d0b8f525d811152edf08e1f6bb2cbabe3f4b52e67538fd652a3195bcf56f9f4a"} Oct 31 03:52:26.394283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.394161 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" event={"ID":"f88f2d72-7161-447d-9b3e-0731e2448c48","Type":"ContainerDied","Data":"11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8"} Oct 31 03:52:26.394283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.394183 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8" Oct 31 03:52:26.394841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.394824 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:52:26.468972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.468878 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:52:26.468972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.468937 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:52:26.493876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.493843 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs\") pod \"f88f2d72-7161-447d-9b3e-0731e2448c48\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " Oct 31 03:52:26.494126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.493894 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj\" (UniqueName: \"kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj\") pod \"f88f2d72-7161-447d-9b3e-0731e2448c48\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " Oct 31 03:52:26.494126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.493936 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca\") pod \"f88f2d72-7161-447d-9b3e-0731e2448c48\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " Oct 31 03:52:26.494126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.494003 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\") pod \"f88f2d72-7161-447d-9b3e-0731e2448c48\" (UID: \"f88f2d72-7161-447d-9b3e-0731e2448c48\") " Oct 31 03:52:26.494469 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.494423 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca" pod "f88f2d72-7161-447d-9b3e-0731e2448c48" (UID: "f88f2d72-7161-447d-9b3e-0731e2448c48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:26.495412 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.495390 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f88f2d72-7161-447d-9b3e-0731e2448c48-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:26.496456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.496431 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs" pod "f88f2d72-7161-447d-9b3e-0731e2448c48" (UID: "f88f2d72-7161-447d-9b3e-0731e2448c48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:26.496800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.496772 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls" pod "f88f2d72-7161-447d-9b3e-0731e2448c48" (UID: "f88f2d72-7161-447d-9b3e-0731e2448c48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:26.496888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.496801 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj" pod "f88f2d72-7161-447d-9b3e-0731e2448c48" (UID: "f88f2d72-7161-447d-9b3e-0731e2448c48"). InnerVolumeSpecName "kube-api-access-sbqsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:26.596968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.596930 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:26.596968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.596969 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqsj\" (UniqueName: \"kubernetes.io/projected/f88f2d72-7161-447d-9b3e-0731e2448c48-kube-api-access-sbqsj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:26.597244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:26.596987 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f88f2d72-7161-447d-9b3e-0731e2448c48-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:27.403027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.402998 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw" Oct 31 03:52:27.443678 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:27.443649 4488 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f2d72_7161_447d_9b3e_0731e2448c48.slice/crio-11343b85356df1739c280014e888737fb0f8be84cd06ed4b2225212fae3bb5a8\": RecentStats: unable to find data in memory cache]" Oct 31 03:52:27.468331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.468293 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:52:27.468485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.468356 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:52:27.529187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.527831 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:27.587308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.587260 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw"] Oct 31 03:52:27.593901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.593868 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2accc9-746964f687-zjcbw"] Oct 31 03:52:27.754171 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.754062 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:27.956434 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:27.956398 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:28.224723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:28.224689 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:28.337027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:28.336994 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88f2d72-7161-447d-9b3e-0731e2448c48" path="/var/lib/kubelet/pods/f88f2d72-7161-447d-9b3e-0731e2448c48/volumes" Oct 31 03:52:28.449215 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:28.449181 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:28.468376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:28.468340 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:52:28.468556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:28.468401 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:52:29.468871 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:29.468839 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:52:29.469292 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:29.468910 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:52:30.468452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:30.468389 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:52:30.468649 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:30.468472 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:52:31.049309 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:31.049271 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs"] Oct 31 03:52:31.067803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:31.067769 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150d759e-xw6vs"] Oct 31 03:52:31.551601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:31.551545 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:31.552344 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:31.552313 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:31.557325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:31.557305 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:52:32.317503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:32.317472 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76d79ee-398d-401c-859e-56030a5b97ad" path="/var/lib/kubelet/pods/c76d79ee-398d-401c-859e-56030a5b97ad/volumes" Oct 31 03:52:33.925302 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:33.925256 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:34.901191 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:34.901161 4488 scope.go:122] "RemoveContainer" containerID="fb2d71ddf22c55ef37b6d3b30082fe9406121e8b1e7c2ea01023ad73b5c78ada" Oct 31 03:52:35.023939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.023900 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:35.181946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.181857 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:35.365762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.365730 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:35.371037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.371007 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9"] Oct 31 03:52:35.371273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.371250 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" containerID="cri-o://c967acdb07235b51de7482b990a342aeffd0f9c153c46f650f4073dab5b4b4c9" gracePeriod=30 Oct 31 03:52:35.533482 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.533358 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:35.858187 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:35.858136 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:36.591224 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:36.591192 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:52:36.712694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:36.712661 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:36.868475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:36.868373 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:38.266044 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:38.266013 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:38.441971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:38.441936 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:39.560392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:39.560358 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:40.365367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.365293 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:40.584719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.584688 4488 generic.go:356] "Generic (PLEG): container finished" podID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerID="c967acdb07235b51de7482b990a342aeffd0f9c153c46f650f4073dab5b4b4c9" exitCode=0 Oct 31 03:52:40.585220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.584745 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" event={"ID":"0c937467-ba00-456f-a0cc-ef9f919855eb","Type":"ContainerDied","Data":"c967acdb07235b51de7482b990a342aeffd0f9c153c46f650f4073dab5b4b4c9"} Oct 31 03:52:40.585220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.584803 4488 scope.go:122] "RemoveContainer" containerID="e67379cc8282000c865d0b26b4c30314a0e7d833ff835595be5cdc1d6d8102fe" Oct 31 03:52:40.665051 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.665024 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:52:40.771601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.771563 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.771809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.771616 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.771809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.771741 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.771942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.771899 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.772017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.771997 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.772017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.771997 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:40.772162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772063 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.772218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772168 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:40.772218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772183 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.772326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772263 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.772326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772273 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:40.772326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772308 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy\") pod \"0c937467-ba00-456f-a0cc-ef9f919855eb\" (UID: \"0c937467-ba00-456f-a0cc-ef9f919855eb\") " Oct 31 03:52:40.772841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772432 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:40.772841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.772641 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:40.773102 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.773053 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.773345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.773101 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c937467-ba00-456f-a0cc-ef9f919855eb-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.773345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.773117 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.773345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.773142 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.773345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.773157 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.774241 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.774216 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:52:40.774370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.774249 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:40.774509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.774493 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "kube-api-access-knvsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:40.774736 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.774720 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy" pod "0c937467-ba00-456f-a0cc-ef9f919855eb" (UID: "0c937467-ba00-456f-a0cc-ef9f919855eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:40.814428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.814383 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv"] Oct 31 03:52:40.825438 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.825405 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk"] Oct 31 03:52:40.834229 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.834198 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj"] Oct 31 03:52:40.875138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.875080 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knvsl\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-kube-api-access-knvsl\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.875138 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.875141 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.875349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.875157 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.875349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.875167 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c937467-ba00-456f-a0cc-ef9f919855eb-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:40.919356 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:40.919256 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:41.593991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:41.593961 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" event={"ID":"0c937467-ba00-456f-a0cc-ef9f919855eb","Type":"ContainerDied","Data":"c9cde71536ca45813c39032839b7c9ad460e70aca4e60c92cd21e56a69cd68cb"} Oct 31 03:52:41.594456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:41.593995 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9" Oct 31 03:52:41.594456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:41.594016 4488 scope.go:122] "RemoveContainer" containerID="c967acdb07235b51de7482b990a342aeffd0f9c153c46f650f4073dab5b4b4c9" Oct 31 03:52:41.760073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:41.760038 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9"] Oct 31 03:52:41.764603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:41.764575 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2accc9-openshift-default-855dc699f7-672f9"] Oct 31 03:52:42.318743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:42.318706 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" path="/var/lib/kubelet/pods/0c937467-ba00-456f-a0cc-ef9f919855eb/volumes" Oct 31 03:52:45.102814 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.102780 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:45.414259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414176 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm"] Oct 31 03:52:45.414769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414755 4488 cpu_manager.go:414] "containerMap: removing container" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerName="payload-pre-processing" Oct 31 03:52:45.414819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414770 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerName="payload-pre-processing" Oct 31 03:52:45.414819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414784 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" Oct 31 03:52:45.414819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414789 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" Oct 31 03:52:45.414819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414802 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f88f2d72-7161-447d-9b3e-0731e2448c48" containerName="maas-api" Oct 31 03:52:45.414819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414807 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88f2d72-7161-447d-9b3e-0731e2448c48" containerName="maas-api" Oct 31 03:52:45.414819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414819 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" Oct 31 03:52:45.414996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414824 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" Oct 31 03:52:45.414996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414954 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e2ddfc-6736-4776-b5f4-c3b7982106b5" containerName="payload-pre-processing" Oct 31 03:52:45.414996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414962 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" Oct 31 03:52:45.414996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414968 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88f2d72-7161-447d-9b3e-0731e2448c48" containerName="maas-api" Oct 31 03:52:45.414996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.414974 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c937467-ba00-456f-a0cc-ef9f919855eb" containerName="istio-proxy" Oct 31 03:52:45.419158 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.419139 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.477321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.477290 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm"] Oct 31 03:52:45.522364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.522330 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:52:45.522364 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.522345 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:52:45.551219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.551187 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.551406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.551385 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.551514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.551427 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ctq\" (UniqueName: \"kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.653334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.653295 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.653536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.653440 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.653536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.653463 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ctq\" (UniqueName: \"kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.654193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.654172 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.655987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.655943 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.715131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.715010 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ctq\" (UniqueName: \"kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq\") pod \"maas-api-revoke-sub-18b37031-wmzhm\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:45.733980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:45.733952 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:46.120103 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:46.120063 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm"] Oct 31 03:52:46.120790 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:52:46.120764 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59cd3c8_ef2c_4786_84aa_bf1bc2682396.slice/crio-40d2de71f362740ed88896ff30fa8d0a9b36d8d0fd1d26c9f467e177310eab94 WatchSource:0}: Error finding container 40d2de71f362740ed88896ff30fa8d0a9b36d8d0fd1d26c9f467e177310eab94: Status 404 returned error can't find the container with id 40d2de71f362740ed88896ff30fa8d0a9b36d8d0fd1d26c9f467e177310eab94 Oct 31 03:52:46.634827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:46.634792 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" event={"ID":"f59cd3c8-ef2c-4786-84aa-bf1bc2682396","Type":"ContainerStarted","Data":"8ac712eeedd7092d56c4193936aad95fedd375d8e55f5365b30ecf9fe4f5111a"} Oct 31 03:52:46.635019 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:46.634852 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" event={"ID":"f59cd3c8-ef2c-4786-84aa-bf1bc2682396","Type":"ContainerStarted","Data":"40d2de71f362740ed88896ff30fa8d0a9b36d8d0fd1d26c9f467e177310eab94"} Oct 31 03:52:46.753549 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:46.753439 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" podStartSLOduration=1.75342579 podStartE2EDuration="1.75342579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:46.749871433 +0000 UTC m=+1994.546589693" watchObservedRunningTime="2026-10-31 03:52:46.75342579 +0000 UTC m=+1994.550144106" Oct 31 03:52:47.643986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:47.643933 4488 generic.go:356] "Generic (PLEG): container finished" podID="f59cd3c8-ef2c-4786-84aa-bf1bc2682396" containerID="8ac712eeedd7092d56c4193936aad95fedd375d8e55f5365b30ecf9fe4f5111a" exitCode=0 Oct 31 03:52:47.644507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:47.644030 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" event={"ID":"f59cd3c8-ef2c-4786-84aa-bf1bc2682396","Type":"ContainerDied","Data":"8ac712eeedd7092d56c4193936aad95fedd375d8e55f5365b30ecf9fe4f5111a"} Oct 31 03:52:48.233891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.233855 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:48.412703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.412651 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:48.819014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.818980 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:48.914366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.914283 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca\") pod \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " Oct 31 03:52:48.914366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.914352 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq\" (UniqueName: \"kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq\") pod \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " Oct 31 03:52:48.914574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.914406 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca\") pod \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\" (UID: \"f59cd3c8-ef2c-4786-84aa-bf1bc2682396\") " Oct 31 03:52:48.914665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.914643 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca" pod "f59cd3c8-ef2c-4786-84aa-bf1bc2682396" (UID: "f59cd3c8-ef2c-4786-84aa-bf1bc2682396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:48.914980 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.914953 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:48.916737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.916716 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca" pod "f59cd3c8-ef2c-4786-84aa-bf1bc2682396" (UID: "f59cd3c8-ef2c-4786-84aa-bf1bc2682396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:48.916867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:48.916849 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq" pod "f59cd3c8-ef2c-4786-84aa-bf1bc2682396" (UID: "f59cd3c8-ef2c-4786-84aa-bf1bc2682396"). InnerVolumeSpecName "kube-api-access-x6ctq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:49.016509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:49.016469 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:49.016509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:49.016503 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ctq\" (UniqueName: \"kubernetes.io/projected/f59cd3c8-ef2c-4786-84aa-bf1bc2682396-kube-api-access-x6ctq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:49.708119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:49.706739 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" event={"ID":"f59cd3c8-ef2c-4786-84aa-bf1bc2682396","Type":"ContainerDied","Data":"40d2de71f362740ed88896ff30fa8d0a9b36d8d0fd1d26c9f467e177310eab94"} Oct 31 03:52:49.708119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:49.706803 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d2de71f362740ed88896ff30fa8d0a9b36d8d0fd1d26c9f467e177310eab94" Oct 31 03:52:49.708119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:49.706870 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm" Oct 31 03:52:50.276880 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:50.276841 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:50.431435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:50.431403 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:50.591772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:50.591742 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:51.791499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:51.791463 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:52.645257 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:52.645223 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:54.373118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:54.372667 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:54.720219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:54.719658 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:55.607723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:55.607683 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:55.840393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:55.840345 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" containerID="cri-o://aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a" gracePeriod=45 Oct 31 03:52:55.847455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:55.847284 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" containerID="cri-o://c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937" gracePeriod=45 Oct 31 03:52:55.855725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:55.855689 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" containerID="cri-o://7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd" gracePeriod=45 Oct 31 03:52:55.911566 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:55.911535 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:52:56.149854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.149649 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:56.259074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.259048 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:52:56.303789 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.303760 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:52:56.305915 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.305894 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:52:56.428379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428281 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj\" (UniqueName: \"kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj\") pod \"d27142dd-64fc-47fc-b2fa-a1105267a78d\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " Oct 31 03:52:56.428379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428352 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir\") pod \"c1475d24-0c69-48d0-a7d3-8357456e1e36\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " Oct 31 03:52:56.428622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428426 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache\") pod \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " Oct 31 03:52:56.428622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428465 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s\" (UniqueName: \"kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s\") pod \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " Oct 31 03:52:56.428622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428508 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home\") pod \"c1475d24-0c69-48d0-a7d3-8357456e1e36\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " Oct 31 03:52:56.428622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428539 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir\") pod \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " Oct 31 03:52:56.428622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428588 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm\") pod \"c1475d24-0c69-48d0-a7d3-8357456e1e36\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " Oct 31 03:52:56.428622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428622 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache\") pod \"c1475d24-0c69-48d0-a7d3-8357456e1e36\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " Oct 31 03:52:56.428943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428667 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir\") pod \"d27142dd-64fc-47fc-b2fa-a1105267a78d\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " Oct 31 03:52:56.428943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428717 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm\") pod \"d27142dd-64fc-47fc-b2fa-a1105267a78d\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " Oct 31 03:52:56.428943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428749 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt\" (UniqueName: \"kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt\") pod \"c1475d24-0c69-48d0-a7d3-8357456e1e36\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " Oct 31 03:52:56.428943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428808 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home\") pod \"d27142dd-64fc-47fc-b2fa-a1105267a78d\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " Oct 31 03:52:56.428943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428849 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs\") pod \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " Oct 31 03:52:56.428943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428887 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm\") pod \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " Oct 31 03:52:56.429335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.428955 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home" pod "c1475d24-0c69-48d0-a7d3-8357456e1e36" (UID: "c1475d24-0c69-48d0-a7d3-8357456e1e36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.429335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429008 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home\") pod \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\" (UID: \"c3d8d2f4-260a-46fc-81e2-f2f39b73597b\") " Oct 31 03:52:56.429335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429058 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache\") pod \"d27142dd-64fc-47fc-b2fa-a1105267a78d\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " Oct 31 03:52:56.429335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429112 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs\") pod \"d27142dd-64fc-47fc-b2fa-a1105267a78d\" (UID: \"d27142dd-64fc-47fc-b2fa-a1105267a78d\") " Oct 31 03:52:56.429335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429181 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir" pod "c1475d24-0c69-48d0-a7d3-8357456e1e36" (UID: "c1475d24-0c69-48d0-a7d3-8357456e1e36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.429335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429187 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs\") pod \"c1475d24-0c69-48d0-a7d3-8357456e1e36\" (UID: \"c1475d24-0c69-48d0-a7d3-8357456e1e36\") " Oct 31 03:52:56.429646 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429410 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache" pod "c3d8d2f4-260a-46fc-81e2-f2f39b73597b" (UID: "c3d8d2f4-260a-46fc-81e2-f2f39b73597b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.429908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429889 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.429986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429910 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.429986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.429919 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.430299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.430278 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir" pod "c3d8d2f4-260a-46fc-81e2-f2f39b73597b" (UID: "c3d8d2f4-260a-46fc-81e2-f2f39b73597b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.431761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.431005 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache" pod "c1475d24-0c69-48d0-a7d3-8357456e1e36" (UID: "c1475d24-0c69-48d0-a7d3-8357456e1e36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.431761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.431316 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home" pod "c3d8d2f4-260a-46fc-81e2-f2f39b73597b" (UID: "c3d8d2f4-260a-46fc-81e2-f2f39b73597b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.431761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.431504 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache" pod "d27142dd-64fc-47fc-b2fa-a1105267a78d" (UID: "d27142dd-64fc-47fc-b2fa-a1105267a78d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.449366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.448269 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home" pod "d27142dd-64fc-47fc-b2fa-a1105267a78d" (UID: "d27142dd-64fc-47fc-b2fa-a1105267a78d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.449366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.448352 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir" pod "d27142dd-64fc-47fc-b2fa-a1105267a78d" (UID: "d27142dd-64fc-47fc-b2fa-a1105267a78d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.449366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.448860 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj" pod "d27142dd-64fc-47fc-b2fa-a1105267a78d" (UID: "d27142dd-64fc-47fc-b2fa-a1105267a78d"). InnerVolumeSpecName "kube-api-access-dvzrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:56.449366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.448944 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s" pod "c3d8d2f4-260a-46fc-81e2-f2f39b73597b" (UID: "c3d8d2f4-260a-46fc-81e2-f2f39b73597b"). InnerVolumeSpecName "kube-api-access-bvh7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:56.449366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.449052 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt" pod "c1475d24-0c69-48d0-a7d3-8357456e1e36" (UID: "c1475d24-0c69-48d0-a7d3-8357456e1e36"). InnerVolumeSpecName "kube-api-access-ghgwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:56.450397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.449961 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm" pod "c3d8d2f4-260a-46fc-81e2-f2f39b73597b" (UID: "c3d8d2f4-260a-46fc-81e2-f2f39b73597b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.450397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.450014 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs" pod "d27142dd-64fc-47fc-b2fa-a1105267a78d" (UID: "d27142dd-64fc-47fc-b2fa-a1105267a78d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:56.450397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.450065 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm" pod "c1475d24-0c69-48d0-a7d3-8357456e1e36" (UID: "c1475d24-0c69-48d0-a7d3-8357456e1e36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.466917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.466876 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs" pod "c1475d24-0c69-48d0-a7d3-8357456e1e36" (UID: "c1475d24-0c69-48d0-a7d3-8357456e1e36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:56.466917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.466876 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs" pod "c3d8d2f4-260a-46fc-81e2-f2f39b73597b" (UID: "c3d8d2f4-260a-46fc-81e2-f2f39b73597b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:52:56.468340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.468308 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm" pod "d27142dd-64fc-47fc-b2fa-a1105267a78d" (UID: "d27142dd-64fc-47fc-b2fa-a1105267a78d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531787 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531827 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531842 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1475d24-0c69-48d0-a7d3-8357456e1e36-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531856 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531870 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531882 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgwt\" (UniqueName: \"kubernetes.io/projected/c1475d24-0c69-48d0-a7d3-8357456e1e36-kube-api-access-ghgwt\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531891 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531903 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531924 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531935 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531963 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27142dd-64fc-47fc-b2fa-a1105267a78d-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531971 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27142dd-64fc-47fc-b2fa-a1105267a78d-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532030 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531979 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1475d24-0c69-48d0-a7d3-8357456e1e36-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.531991 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzrj\" (UniqueName: \"kubernetes.io/projected/d27142dd-64fc-47fc-b2fa-a1105267a78d-kube-api-access-dvzrj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.532827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.532689 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvh7s\" (UniqueName: \"kubernetes.io/projected/c3d8d2f4-260a-46fc-81e2-f2f39b73597b-kube-api-access-bvh7s\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:52:56.766759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.766661 4488 generic.go:356] "Generic (PLEG): container finished" podID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerID="7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd" exitCode=0 Oct 31 03:52:56.766759 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.766722 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" Oct 31 03:52:56.767362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.766761 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" event={"ID":"c3d8d2f4-260a-46fc-81e2-f2f39b73597b","Type":"ContainerDied","Data":"7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd"} Oct 31 03:52:56.767362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.766800 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj" event={"ID":"c3d8d2f4-260a-46fc-81e2-f2f39b73597b","Type":"ContainerDied","Data":"15f543af0b5b60154c3b826801ad415d194961be654503745a1fd19026ab199a"} Oct 31 03:52:56.767362 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.766820 4488 scope.go:122] "RemoveContainer" containerID="7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd" Oct 31 03:52:56.768769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.768734 4488 generic.go:356] "Generic (PLEG): container finished" podID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerID="aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a" exitCode=0 Oct 31 03:52:56.768891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.768845 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" event={"ID":"d27142dd-64fc-47fc-b2fa-a1105267a78d","Type":"ContainerDied","Data":"aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a"} Oct 31 03:52:56.768891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.768878 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" event={"ID":"d27142dd-64fc-47fc-b2fa-a1105267a78d","Type":"ContainerDied","Data":"28e5e905682c0240322dbcb88fc7351d61f749d117ee50a2a6c7ce4988450823"} Oct 31 03:52:56.768997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.768950 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv" Oct 31 03:52:56.772778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.772744 4488 generic.go:356] "Generic (PLEG): container finished" podID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerID="c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937" exitCode=0 Oct 31 03:52:56.772906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.772819 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" event={"ID":"c1475d24-0c69-48d0-a7d3-8357456e1e36","Type":"ContainerDied","Data":"c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937"} Oct 31 03:52:56.772906 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.772861 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" event={"ID":"c1475d24-0c69-48d0-a7d3-8357456e1e36","Type":"ContainerDied","Data":"75191c2697e72259c89a296c7ede5f190de3f5ca3b67799a4b4b1aeec49a1b3e"} Oct 31 03:52:56.772999 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.772910 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" Oct 31 03:52:56.779610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.779588 4488 scope.go:122] "RemoveContainer" containerID="7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd" Oct 31 03:52:56.779972 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:56.779929 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd\": container with ID starting with 7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd not found: ID does not exist" containerID="7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd" Oct 31 03:52:56.780065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.779982 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd"} err="failed to get container status \"7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd\": rpc error: code = NotFound desc = could not find container \"7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd\": container with ID starting with 7cb6db1803167b35568d30d7e162f96b1c050c1480c9e16425679147b7ac4cdd not found: ID does not exist" Oct 31 03:52:56.780065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.780000 4488 scope.go:122] "RemoveContainer" containerID="aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a" Oct 31 03:52:56.790971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.790956 4488 scope.go:122] "RemoveContainer" containerID="aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a" Oct 31 03:52:56.791279 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:56.791257 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a\": container with ID starting with aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a not found: ID does not exist" containerID="aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a" Oct 31 03:52:56.791334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.791287 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a"} err="failed to get container status \"aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a\": rpc error: code = NotFound desc = could not find container \"aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a\": container with ID starting with aa7916843f215b61c263bc54a1f0ca70042b11732b20a0835ca9ab332e4a8b4a not found: ID does not exist" Oct 31 03:52:56.791334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.791305 4488 scope.go:122] "RemoveContainer" containerID="c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937" Oct 31 03:52:56.800939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.800918 4488 scope.go:122] "RemoveContainer" containerID="c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937" Oct 31 03:52:56.801283 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:52:56.801260 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937\": container with ID starting with c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937 not found: ID does not exist" containerID="c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937" Oct 31 03:52:56.801341 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.801294 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937"} err="failed to get container status \"c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937\": rpc error: code = NotFound desc = could not find container \"c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937\": container with ID starting with c1f93077e2a57f577ec597b31d284d9c42e39256efb6ae646e6dcd7af8a34937 not found: ID does not exist" Oct 31 03:52:56.934947 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.934911 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj"] Oct 31 03:52:56.940183 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:56.940137 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/simulator-w1-e00ad3-kserve-6fbff9cb9c-hs8qj"] Oct 31 03:52:57.046415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:57.046315 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv"] Oct 31 03:52:57.052813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:57.052776 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-w1-e00ad3-kserve-75c64b4d6c-kcshv"] Oct 31 03:52:57.161466 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:57.161430 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk"] Oct 31 03:52:57.165614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:57.165586 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk"] Oct 31 03:52:57.280034 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:57.279987 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/premium-w1-e00ad3-kserve-6d495cb544-rpsmk" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:58.325115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:58.324774 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" path="/var/lib/kubelet/pods/c1475d24-0c69-48d0-a7d3-8357456e1e36/volumes" Oct 31 03:52:58.325657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:58.325530 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" path="/var/lib/kubelet/pods/c3d8d2f4-260a-46fc-81e2-f2f39b73597b/volumes" Oct 31 03:52:58.327139 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:58.326177 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" path="/var/lib/kubelet/pods/d27142dd-64fc-47fc-b2fa-a1105267a78d/volumes" Oct 31 03:52:58.678119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:58.673027 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:58.884116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:58.883430 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:59.106114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:59.079761 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:59.696116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:59.694203 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:59.708114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:59.705192 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:59.730117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:59.721237 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:59.762137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:52:59.761250 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.364115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.362185 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 31 03:53:00.492300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.492241 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.564529 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.564471 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.580232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.580170 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.594224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.594178 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.605233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.605188 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:53:00.621405 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.621301 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.699562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.699521 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:00.700488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.700468 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:53:00.773940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.769720 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" Oct 31 03:53:00.951968 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:00.951865 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:02.132223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:02.132192 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:02.302756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:02.302686 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:02.921350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:02.921319 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:06.588801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:06.588761 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:06.592547 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:53:06.592515 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:53:10.522006 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:10.521963 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:11.841248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:11.841212 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:17.322999 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:17.322885 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:17.497639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:17.497605 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:18.367681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:18.367648 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:20.247024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:20.246986 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:22.082827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:22.082790 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:22.243729 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:22.243692 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:22.402478 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:22.402390 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:26.340564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:26.340535 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:26.497367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:26.497337 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:27.082625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:27.082593 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:28.568802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:28.568769 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:30.637824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:30.637798 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:30.791605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:30.791575 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:31.911561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:31.911514 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:34.946985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:34.946951 4488 scope.go:122] "RemoveContainer" containerID="4682db2ca5c2526c6c7cd836a5a9c338fb76fa3d25cbf1b953cc3b185b2c179b" Oct 31 03:53:34.957031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:34.957008 4488 scope.go:122] "RemoveContainer" containerID="0a10fc13538260c4d15b7a7abb50fd45580b9bbb489964cc8eb97017d217568e" Oct 31 03:53:36.591613 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:53:36.591563 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:53:36.719067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:36.719032 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:41.869618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:41.869571 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:42.213299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.213192 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.275243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.275189 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.301198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.298245 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.536313 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.536205 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.604115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.600307 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.604440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.604400 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.604958 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.604601 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.616210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.616170 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.625284 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.625245 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.678626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.678574 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.728128 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.726229 4488 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": context deadline exceeded" Oct 31 03:53:42.765576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.765517 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:42.780637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:42.780588 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:46.744887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:46.744852 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:51.578889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:51.578847 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:56.407145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:56.407083 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:58.506291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:58.506254 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:58.667451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:53:58.667417 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:00.579516 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.579485 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6"] Oct 31 03:54:00.580015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.579682 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" podUID="64d269d0-254a-4a11-8f72-68c8a060deb7" containerName="maas-api" containerID="cri-o://f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67" gracePeriod=30 Oct 31 03:54:00.690738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.690707 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:00.896192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.896169 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:54:00.990119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.989987 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs\") pod \"64d269d0-254a-4a11-8f72-68c8a060deb7\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " Oct 31 03:54:00.993111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.993058 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\") pod \"64d269d0-254a-4a11-8f72-68c8a060deb7\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " Oct 31 03:54:00.993299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.993179 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca\") pod \"64d269d0-254a-4a11-8f72-68c8a060deb7\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " Oct 31 03:54:00.993299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.993233 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6\" (UniqueName: \"kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6\") pod \"64d269d0-254a-4a11-8f72-68c8a060deb7\" (UID: \"64d269d0-254a-4a11-8f72-68c8a060deb7\") " Oct 31 03:54:00.994561 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:00.994503 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca" pod "64d269d0-254a-4a11-8f72-68c8a060deb7" (UID: "64d269d0-254a-4a11-8f72-68c8a060deb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:01.001342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.001236 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls" pod "64d269d0-254a-4a11-8f72-68c8a060deb7" (UID: "64d269d0-254a-4a11-8f72-68c8a060deb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:01.001342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.001262 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs" pod "64d269d0-254a-4a11-8f72-68c8a060deb7" (UID: "64d269d0-254a-4a11-8f72-68c8a060deb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:01.004018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.003989 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6" pod "64d269d0-254a-4a11-8f72-68c8a060deb7" (UID: "64d269d0-254a-4a11-8f72-68c8a060deb7"). InnerVolumeSpecName "kube-api-access-v5vp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:01.095653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.095618 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.095653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.095649 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d269d0-254a-4a11-8f72-68c8a060deb7-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.095653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.095660 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5vp6\" (UniqueName: \"kubernetes.io/projected/64d269d0-254a-4a11-8f72-68c8a060deb7-kube-api-access-v5vp6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.095914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.095671 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d269d0-254a-4a11-8f72-68c8a060deb7-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:01.656112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.641058 4488 generic.go:356] "Generic (PLEG): container finished" podID="64d269d0-254a-4a11-8f72-68c8a060deb7" containerID="f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67" exitCode=0 Oct 31 03:54:01.656112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.641363 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" event={"ID":"64d269d0-254a-4a11-8f72-68c8a060deb7","Type":"ContainerDied","Data":"f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67"} Oct 31 03:54:01.656112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.641418 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" event={"ID":"64d269d0-254a-4a11-8f72-68c8a060deb7","Type":"ContainerDied","Data":"c7a70d18bacebcb62fcf1d027ade54c02b84a0ebc27c6c7fd7d5934b6881bc1d"} Oct 31 03:54:01.656112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.641448 4488 scope.go:122] "RemoveContainer" containerID="f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67" Oct 31 03:54:01.656112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.641616 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6" Oct 31 03:54:01.677680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.677656 4488 scope.go:122] "RemoveContainer" containerID="f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67" Oct 31 03:54:01.679143 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:01.679114 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67\": container with ID starting with f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67 not found: ID does not exist" containerID="f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67" Oct 31 03:54:01.679288 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.679152 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67"} err="failed to get container status \"f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67\": rpc error: code = NotFound desc = could not find container \"f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67\": container with ID starting with f2ff63f2d1de224b82cd394a860e5a7b8f8f5a08144057fb1a8c3217517dda67 not found: ID does not exist" Oct 31 03:54:01.816741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.816701 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6"] Oct 31 03:54:01.866754 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:01.866715 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-2dvd6"] Oct 31 03:54:02.230761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:02.230726 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:02.318627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:02.318593 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d269d0-254a-4a11-8f72-68c8a060deb7" path="/var/lib/kubelet/pods/64d269d0-254a-4a11-8f72-68c8a060deb7/volumes" Oct 31 03:54:02.509608 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:02.509525 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:02.665743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:02.665712 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:04.320476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:04.320436 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:04.479754 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:04.479725 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:06.592871 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:06.592831 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:54:06.767099 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:06.767055 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:07.659738 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:07.659703 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:10.294256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:10.294217 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:10.461800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:10.461768 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:12.613121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:12.613054 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:12.796879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:12.796730 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:13.014656 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:13.014564 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:16.755420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:16.755385 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:17.833129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.833071 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65"] Oct 31 03:54:17.833978 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.833956 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" Oct 31 03:54:17.834105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.833979 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" Oct 31 03:54:17.834105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.833998 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" Oct 31 03:54:17.834105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834006 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" Oct 31 03:54:17.834105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834064 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" Oct 31 03:54:17.834105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834073 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" Oct 31 03:54:17.834105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834102 4488 cpu_manager.go:414] "containerMap: removing container" podUID="64d269d0-254a-4a11-8f72-68c8a060deb7" containerName="maas-api" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834110 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d269d0-254a-4a11-8f72-68c8a060deb7" containerName="maas-api" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834131 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f59cd3c8-ef2c-4786-84aa-bf1bc2682396" containerName="revoke-keys" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834138 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59cd3c8-ef2c-4786-84aa-bf1bc2682396" containerName="revoke-keys" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834307 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1475d24-0c69-48d0-a7d3-8357456e1e36" containerName="main" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834321 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d269d0-254a-4a11-8f72-68c8a060deb7" containerName="maas-api" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834335 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59cd3c8-ef2c-4786-84aa-bf1bc2682396" containerName="revoke-keys" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834346 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d8d2f4-260a-46fc-81e2-f2f39b73597b" containerName="main" Oct 31 03:54:17.834393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.834356 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27142dd-64fc-47fc-b2fa-a1105267a78d" containerName="main" Oct 31 03:54:17.839458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.839440 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:17.892153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.892061 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b16de9-serving-cert\"" Oct 31 03:54:17.909260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.909220 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzm6\" (UniqueName: \"kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:17.909422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.909389 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:17.909479 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.909455 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:17.909571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.909555 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:17.946875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:17.946840 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65"] Oct 31 03:54:18.010679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.010644 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.010856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.010727 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.010856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.010781 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzm6\" (UniqueName: \"kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.010945 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.010873 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.011056 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:18.011032 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-serving-cert: secret "maas-api-e2e-api-b-b16de9-serving-cert" not found Oct 31 03:54:18.011144 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:18.011130 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls podName:c620b32c-0e8a-4118-92f9-275b62e09266 nodeName:}" failed. No retries permitted until 2026-10-31 03:54:18.51110837 +0000 UTC m=+2086.307826612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls") pod "maas-api-e2e-api-b-b16de9-597b896f9-g5z65" (UID: "c620b32c-0e8a-4118-92f9-275b62e09266") : secret "maas-api-e2e-api-b-b16de9-serving-cert" not found Oct 31 03:54:18.011542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.011524 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.013310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.013294 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.029884 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.029857 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.096149 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.095984 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzm6\" (UniqueName: \"kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.211014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.210985 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.376728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.376639 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.520503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.520461 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.523142 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.523108 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\") pod \"maas-api-e2e-api-b-b16de9-597b896f9-g5z65\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:18.752943 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:18.752844 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:19.140129 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:19.140078 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65"] Oct 31 03:54:19.141386 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:54:19.141356 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc620b32c_0e8a_4118_92f9_275b62e09266.slice/crio-b816ed7723e016d84f4fdc2c5429892839fa29e012f76b9ef08b0a3f2179e80e WatchSource:0}: Error finding container b816ed7723e016d84f4fdc2c5429892839fa29e012f76b9ef08b0a3f2179e80e: Status 404 returned error can't find the container with id b816ed7723e016d84f4fdc2c5429892839fa29e012f76b9ef08b0a3f2179e80e Oct 31 03:54:19.804678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:19.804642 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" event={"ID":"c620b32c-0e8a-4118-92f9-275b62e09266","Type":"ContainerStarted","Data":"b816ed7723e016d84f4fdc2c5429892839fa29e012f76b9ef08b0a3f2179e80e"} Oct 31 03:54:20.812614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:20.812570 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" event={"ID":"c620b32c-0e8a-4118-92f9-275b62e09266","Type":"ContainerStarted","Data":"0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e"} Oct 31 03:54:20.813143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:20.812633 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:20.937122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:20.937051 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" podStartSLOduration=3.109335936 podStartE2EDuration="3.93703516s" totalImagesPullingTime="827.699224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:20.935053088 +0000 UTC m=+2088.731771345" watchObservedRunningTime="2026-10-31 03:54:20.93703516 +0000 UTC m=+2088.733753418" Oct 31 03:54:22.172275 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:22.172235 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:54:23.510069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:23.510033 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:26.868625 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:26.868595 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:54:28.715483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:28.715446 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:29.905398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:29.905365 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js"] Oct 31 03:54:30.016837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:30.016801 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7"] Oct 31 03:54:30.184004 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:30.183896 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:30.363063 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:30.361274 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:30.520945 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:30.520860 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:33.239550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:33.239515 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:33.440724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:33.440692 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:33.720767 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:33.720732 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:34.905699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:34.905645 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" podUID="95d9272b-bc18-4749-9818-cd6701024fd3" containerName="payload-processing" containerID="cri-o://4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649" gracePeriod=25 Oct 31 03:54:35.010308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.010276 4488 scope.go:122] "RemoveContainer" containerID="e32a9bc563fbf2a52c47d4300359bfd97a4b6bc9eb09fde4a9a034fd6525caa3" Oct 31 03:54:35.017514 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.017474 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" podUID="2273bfb2-857f-416f-82da-9550eb28a682" containerName="payload-pre-processing" containerID="cri-o://a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0" gracePeriod=25 Oct 31 03:54:35.213460 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.213422 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:54:35.288957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.288923 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume\") pod \"95d9272b-bc18-4749-9818-cd6701024fd3\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " Oct 31 03:54:35.289131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.289007 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5\" (UniqueName: \"kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5\") pod \"95d9272b-bc18-4749-9818-cd6701024fd3\" (UID: \"95d9272b-bc18-4749-9818-cd6701024fd3\") " Oct 31 03:54:35.289370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.289345 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume" pod "95d9272b-bc18-4749-9818-cd6701024fd3" (UID: "95d9272b-bc18-4749-9818-cd6701024fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:35.289671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.289654 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d9272b-bc18-4749-9818-cd6701024fd3-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:35.291415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.291391 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5" pod "95d9272b-bc18-4749-9818-cd6701024fd3" (UID: "95d9272b-bc18-4749-9818-cd6701024fd3"). InnerVolumeSpecName "kube-api-access-vz8z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:35.298601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.298580 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:54:35.390833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.390797 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume\") pod \"2273bfb2-857f-416f-82da-9550eb28a682\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " Oct 31 03:54:35.391028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.390877 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj\" (UniqueName: \"kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj\") pod \"2273bfb2-857f-416f-82da-9550eb28a682\" (UID: \"2273bfb2-857f-416f-82da-9550eb28a682\") " Oct 31 03:54:35.391299 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.391274 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume" pod "2273bfb2-857f-416f-82da-9550eb28a682" (UID: "2273bfb2-857f-416f-82da-9550eb28a682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:35.391583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.391563 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz8z5\" (UniqueName: \"kubernetes.io/projected/95d9272b-bc18-4749-9818-cd6701024fd3-kube-api-access-vz8z5\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:35.391583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.391584 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2273bfb2-857f-416f-82da-9550eb28a682-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:35.393327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.393308 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj" pod "2273bfb2-857f-416f-82da-9550eb28a682" (UID: "2273bfb2-857f-416f-82da-9550eb28a682"). InnerVolumeSpecName "kube-api-access-68qcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:35.493170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.493051 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qcj\" (UniqueName: \"kubernetes.io/projected/2273bfb2-857f-416f-82da-9550eb28a682-kube-api-access-68qcj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:54:35.961681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.961648 4488 generic.go:356] "Generic (PLEG): container finished" podID="95d9272b-bc18-4749-9818-cd6701024fd3" containerID="4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649" exitCode=0 Oct 31 03:54:35.962201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.961699 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" Oct 31 03:54:35.962201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.961723 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" event={"ID":"95d9272b-bc18-4749-9818-cd6701024fd3","Type":"ContainerDied","Data":"4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649"} Oct 31 03:54:35.962201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.961792 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js" event={"ID":"95d9272b-bc18-4749-9818-cd6701024fd3","Type":"ContainerDied","Data":"3fd67bdc65ca85063539fda5ac0018bb62c25550453d25a60d887503bac01dc0"} Oct 31 03:54:35.962201 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.961820 4488 scope.go:122] "RemoveContainer" containerID="4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649" Oct 31 03:54:35.963232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.963103 4488 generic.go:356] "Generic (PLEG): container finished" podID="2273bfb2-857f-416f-82da-9550eb28a682" containerID="a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0" exitCode=0 Oct 31 03:54:35.963232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.963162 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" Oct 31 03:54:35.963232 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.963210 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" event={"ID":"2273bfb2-857f-416f-82da-9550eb28a682","Type":"ContainerDied","Data":"a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0"} Oct 31 03:54:35.963379 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.963263 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7" event={"ID":"2273bfb2-857f-416f-82da-9550eb28a682","Type":"ContainerDied","Data":"23ee54f4939a2d846c0221a318eb74bf34beafb7ab6ede3e662a968a460b5e58"} Oct 31 03:54:35.972495 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.972406 4488 scope.go:122] "RemoveContainer" containerID="4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649" Oct 31 03:54:35.972752 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:35.972727 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649\": container with ID starting with 4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649 not found: ID does not exist" containerID="4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649" Oct 31 03:54:35.972875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.972761 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649"} err="failed to get container status \"4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649\": rpc error: code = NotFound desc = could not find container \"4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649\": container with ID starting with 4975d48b4c26b0265eddbf42114af3be1089ab8faf5ab5dec7f82bd41cd0b649 not found: ID does not exist" Oct 31 03:54:35.972875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.972784 4488 scope.go:122] "RemoveContainer" containerID="a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0" Oct 31 03:54:35.983393 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.983367 4488 scope.go:122] "RemoveContainer" containerID="a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0" Oct 31 03:54:35.983729 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:35.983706 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0\": container with ID starting with a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0 not found: ID does not exist" containerID="a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0" Oct 31 03:54:35.983780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:35.983738 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0"} err="failed to get container status \"a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0\": rpc error: code = NotFound desc = could not find container \"a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0\": container with ID starting with a008ef974c5dd9c579a5dfb5ae30e176d5f12e31c085a79a61390f675c5aa2f0 not found: ID does not exist" Oct 31 03:54:36.128147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:36.128083 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7"] Oct 31 03:54:36.177946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:36.177863 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc91a3-5b7fbf9b79-h2zg7"] Oct 31 03:54:36.243626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:36.243592 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js"] Oct 31 03:54:36.251458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:36.251421 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc91a3-5889cc7877-854js"] Oct 31 03:54:36.317720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:36.317690 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2273bfb2-857f-416f-82da-9550eb28a682" path="/var/lib/kubelet/pods/2273bfb2-857f-416f-82da-9550eb28a682/volumes" Oct 31 03:54:36.318148 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:36.318134 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d9272b-bc18-4749-9818-cd6701024fd3" path="/var/lib/kubelet/pods/95d9272b-bc18-4749-9818-cd6701024fd3/volumes" Oct 31 03:54:36.590921 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:54:36.590874 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:54:37.536376 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:37.536345 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:38.502285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:38.502246 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:42.767720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:42.767683 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:44.106761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:44.106658 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:44.282703 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:44.282652 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:44.500583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:44.500479 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:44.700772 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:44.700729 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:45.263712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:45.263674 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:49.740492 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:49.740460 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:52.052153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:52.052120 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:52.240615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:52.240552 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:52.391925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:52.391875 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:53.167733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:53.167696 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:54.997853 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:54.997823 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:59.905819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:54:59.905782 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:01.559661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:01.559614 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:55:04.587116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:04.586200 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:55:05.007355 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:05.007260 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:06.591431 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:55:06.591394 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:55:06.639427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:06.639397 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4"] Oct 31 03:55:06.747882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:06.747847 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55"] Oct 31 03:55:07.007721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:07.007634 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:07.124156 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:07.124122 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb"] Oct 31 03:55:07.128407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:07.128368 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74"] Oct 31 03:55:07.131372 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:07.131342 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80ae1e5-5ww74"] Oct 31 03:55:07.134659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:07.134627 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2b4bfc-2l2fb"] Oct 31 03:55:07.163922 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:07.163882 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:08.338519 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:08.338478 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbdd5d5-1df0-4c59-8603-4c554162332f" path="/var/lib/kubelet/pods/2bbdd5d5-1df0-4c59-8603-4c554162332f/volumes" Oct 31 03:55:08.346345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:08.346309 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775baf45-b922-4333-94a6-a0c03c46d84c" path="/var/lib/kubelet/pods/775baf45-b922-4333-94a6-a0c03c46d84c/volumes" Oct 31 03:55:09.791306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:09.791278 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:11.640371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:11.640299 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" podUID="3db94ead-53db-45ff-9735-a53dd2a0c38b" containerName="payload-processing" containerID="cri-o://a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e" gracePeriod=25 Oct 31 03:55:11.749253 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:11.749211 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" podUID="bfb32f56-d27d-4e57-a006-306de73598f8" containerName="payload-pre-processing" containerID="cri-o://81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f" gracePeriod=25 Oct 31 03:55:11.931388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:11.931358 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:55:12.003332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.003300 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn\" (UniqueName: \"kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn\") pod \"3db94ead-53db-45ff-9735-a53dd2a0c38b\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " Oct 31 03:55:12.003609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.003591 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume\") pod \"3db94ead-53db-45ff-9735-a53dd2a0c38b\" (UID: \"3db94ead-53db-45ff-9735-a53dd2a0c38b\") " Oct 31 03:55:12.004067 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.004038 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume" pod "3db94ead-53db-45ff-9735-a53dd2a0c38b" (UID: "3db94ead-53db-45ff-9735-a53dd2a0c38b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:12.006108 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.006067 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn" pod "3db94ead-53db-45ff-9735-a53dd2a0c38b" (UID: "3db94ead-53db-45ff-9735-a53dd2a0c38b"). InnerVolumeSpecName "kube-api-access-26zbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:12.025457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.025427 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:55:12.105199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.105150 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5\" (UniqueName: \"kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5\") pod \"bfb32f56-d27d-4e57-a006-306de73598f8\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " Oct 31 03:55:12.105406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.105225 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume\") pod \"bfb32f56-d27d-4e57-a006-306de73598f8\" (UID: \"bfb32f56-d27d-4e57-a006-306de73598f8\") " Oct 31 03:55:12.105695 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.105665 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume" pod "bfb32f56-d27d-4e57-a006-306de73598f8" (UID: "bfb32f56-d27d-4e57-a006-306de73598f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:12.105847 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.105831 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db94ead-53db-45ff-9735-a53dd2a0c38b-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:55:12.105925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.105849 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb32f56-d27d-4e57-a006-306de73598f8-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:55:12.105925 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.105859 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zbn\" (UniqueName: \"kubernetes.io/projected/3db94ead-53db-45ff-9735-a53dd2a0c38b-kube-api-access-26zbn\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:55:12.107802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.107777 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5" pod "bfb32f56-d27d-4e57-a006-306de73598f8" (UID: "bfb32f56-d27d-4e57-a006-306de73598f8"). InnerVolumeSpecName "kube-api-access-vctd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:12.208639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.208390 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctd5\" (UniqueName: \"kubernetes.io/projected/bfb32f56-d27d-4e57-a006-306de73598f8-kube-api-access-vctd5\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:55:12.283800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.283768 4488 generic.go:356] "Generic (PLEG): container finished" podID="3db94ead-53db-45ff-9735-a53dd2a0c38b" containerID="a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e" exitCode=0 Oct 31 03:55:12.284031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.283834 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" Oct 31 03:55:12.284031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.283853 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" event={"ID":"3db94ead-53db-45ff-9735-a53dd2a0c38b","Type":"ContainerDied","Data":"a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e"} Oct 31 03:55:12.284031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.283972 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4" event={"ID":"3db94ead-53db-45ff-9735-a53dd2a0c38b","Type":"ContainerDied","Data":"3063f3862143bb1998c4ea7428677826637ebf3acd4887b0bdbf95327b6a5797"} Oct 31 03:55:12.284031 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.284005 4488 scope.go:122] "RemoveContainer" containerID="a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e" Oct 31 03:55:12.285526 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.285499 4488 generic.go:356] "Generic (PLEG): container finished" podID="bfb32f56-d27d-4e57-a006-306de73598f8" containerID="81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f" exitCode=0 Oct 31 03:55:12.285647 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.285568 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" Oct 31 03:55:12.285700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.285568 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" event={"ID":"bfb32f56-d27d-4e57-a006-306de73598f8","Type":"ContainerDied","Data":"81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f"} Oct 31 03:55:12.285787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.285713 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55" event={"ID":"bfb32f56-d27d-4e57-a006-306de73598f8","Type":"ContainerDied","Data":"bc30280b7407662e3d2876a706d7ffbad931e81ab131f38c88834050d3398006"} Oct 31 03:55:12.297169 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.297141 4488 scope.go:122] "RemoveContainer" containerID="a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e" Oct 31 03:55:12.297541 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:55:12.297518 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e\": container with ID starting with a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e not found: ID does not exist" containerID="a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e" Oct 31 03:55:12.297595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.297552 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e"} err="failed to get container status \"a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e\": rpc error: code = NotFound desc = could not find container \"a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e\": container with ID starting with a6860467531fd04aca2db4320a1b4894af3bc595d3b4d825085d014e214a697e not found: ID does not exist" Oct 31 03:55:12.297595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.297570 4488 scope.go:122] "RemoveContainer" containerID="81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f" Oct 31 03:55:12.307510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.307488 4488 scope.go:122] "RemoveContainer" containerID="81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f" Oct 31 03:55:12.307838 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:55:12.307816 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f\": container with ID starting with 81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f not found: ID does not exist" containerID="81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f" Oct 31 03:55:12.307896 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.307847 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f"} err="failed to get container status \"81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f\": rpc error: code = NotFound desc = could not find container \"81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f\": container with ID starting with 81db0b243b447d0b2bf2067bbfe98e93f2a7bc7cf135d59000e7c24aa192357f not found: ID does not exist" Oct 31 03:55:12.454069 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.454010 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55"] Oct 31 03:55:12.459046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.458964 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c43a6-6b87d5d675-r8q55"] Oct 31 03:55:12.567595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.567558 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4"] Oct 31 03:55:12.575680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:12.575647 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c43a6-7f74dc657f-8dxv4"] Oct 31 03:55:14.317132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:14.317080 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db94ead-53db-45ff-9735-a53dd2a0c38b" path="/var/lib/kubelet/pods/3db94ead-53db-45ff-9735-a53dd2a0c38b/volumes" Oct 31 03:55:14.317756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:14.317739 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb32f56-d27d-4e57-a006-306de73598f8" path="/var/lib/kubelet/pods/bfb32f56-d27d-4e57-a006-306de73598f8/volumes" Oct 31 03:55:14.556966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:14.556936 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.079773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.079740 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc"] Oct 31 03:55:19.080445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080427 4488 cpu_manager.go:414] "containerMap: removing container" podUID="95d9272b-bc18-4749-9818-cd6701024fd3" containerName="payload-processing" Oct 31 03:55:19.080497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080447 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d9272b-bc18-4749-9818-cd6701024fd3" containerName="payload-processing" Oct 31 03:55:19.080497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080487 4488 cpu_manager.go:414] "containerMap: removing container" podUID="bfb32f56-d27d-4e57-a006-306de73598f8" containerName="payload-pre-processing" Oct 31 03:55:19.080497 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080492 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb32f56-d27d-4e57-a006-306de73598f8" containerName="payload-pre-processing" Oct 31 03:55:19.080596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080504 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3db94ead-53db-45ff-9735-a53dd2a0c38b" containerName="payload-processing" Oct 31 03:55:19.080596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080509 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db94ead-53db-45ff-9735-a53dd2a0c38b" containerName="payload-processing" Oct 31 03:55:19.080596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080519 4488 cpu_manager.go:414] "containerMap: removing container" podUID="2273bfb2-857f-416f-82da-9550eb28a682" containerName="payload-pre-processing" Oct 31 03:55:19.080596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080524 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2273bfb2-857f-416f-82da-9550eb28a682" containerName="payload-pre-processing" Oct 31 03:55:19.080800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080607 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb32f56-d27d-4e57-a006-306de73598f8" containerName="payload-pre-processing" Oct 31 03:55:19.080800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080615 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db94ead-53db-45ff-9735-a53dd2a0c38b" containerName="payload-processing" Oct 31 03:55:19.080800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080623 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="2273bfb2-857f-416f-82da-9550eb28a682" containerName="payload-pre-processing" Oct 31 03:55:19.080800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.080631 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d9272b-bc18-4749-9818-cd6701024fd3" containerName="payload-processing" Oct 31 03:55:19.085126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.085081 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.139024 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.138990 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1180d818-serving-cert\"" Oct 31 03:55:19.143244 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.143219 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc"] Oct 31 03:55:19.205866 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.205825 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.206050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.205886 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.206050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.206013 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt58l\" (UniqueName: \"kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.206050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.206047 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.281423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.281385 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.306986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.306954 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.307194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.307006 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.307194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.307046 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt58l\" (UniqueName: \"kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.307194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.307063 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.307340 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:55:19.307186 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-serving-cert: secret "maas-api-e2e-mt-1180d818-serving-cert" not found Oct 31 03:55:19.307340 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:55:19.307282 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls podName:0fee47dc-7209-46db-8f3c-a27b016a114f nodeName:}" failed. No retries permitted until 2026-10-31 03:55:19.807258428 +0000 UTC m=+2147.603976666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls") pod "maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" (UID: "0fee47dc-7209-46db-8f3c-a27b016a114f") : secret "maas-api-e2e-mt-1180d818-serving-cert" not found Oct 31 03:55:19.307790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.307768 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.309827 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.309803 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.372477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.371265 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt58l\" (UniqueName: \"kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.443388 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.443355 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.600774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.600736 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.815011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.814929 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.817645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.817624 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\") pod \"maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:19.998618 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:19.998574 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:20.379203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:20.379176 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc"] Oct 31 03:55:20.380211 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:55:20.380184 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fee47dc_7209_46db_8f3c_a27b016a114f.slice/crio-90281fecdfab99e8ca3ff54fa05b215d5c46da852dc708e988c176861a157b9d WatchSource:0}: Error finding container 90281fecdfab99e8ca3ff54fa05b215d5c46da852dc708e988c176861a157b9d: Status 404 returned error can't find the container with id 90281fecdfab99e8ca3ff54fa05b215d5c46da852dc708e988c176861a157b9d Oct 31 03:55:20.381654 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:20.381637 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:55:21.374038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:21.374004 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" event={"ID":"0fee47dc-7209-46db-8f3c-a27b016a114f","Type":"ContainerStarted","Data":"deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90"} Oct 31 03:55:21.374243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:21.374065 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" event={"ID":"0fee47dc-7209-46db-8f3c-a27b016a114f","Type":"ContainerStarted","Data":"90281fecdfab99e8ca3ff54fa05b215d5c46da852dc708e988c176861a157b9d"} Oct 31 03:55:21.374243 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:21.374155 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:21.513060 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:21.512982 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" podStartSLOduration=1.617833656 podStartE2EDuration="2.512967061s" totalImagesPullingTime="895.133405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:21.511062711 +0000 UTC m=+2149.307780970" watchObservedRunningTime="2026-10-31 03:55:21.512967061 +0000 UTC m=+2149.309685319" Oct 31 03:55:24.310756 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:24.310711 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:27.386833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:27.386804 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:55:29.108624 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:29.108583 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:29.261697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:29.261667 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:33.880859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:33.880813 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:35.154758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:35.154726 4488 scope.go:122] "RemoveContainer" containerID="d6560d89dd3addac59d5a02f1a0582b386d985cf304c660ca13ba065c290d097" Oct 31 03:55:35.164914 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:35.164891 4488 scope.go:122] "RemoveContainer" containerID="d0b8f525d811152edf08e1f6bb2cbabe3f4b52e67538fd652a3195bcf56f9f4a" Oct 31 03:55:36.593304 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:55:36.593261 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:55:38.615691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:38.615656 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:55:38.644938 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:38.644905 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:43.517700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:43.517659 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:47.648904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:47.648869 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:48.732764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:48.732711 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:53.905668 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:53.905606 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:59.299152 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:55:59.299113 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:02.066614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:02.066583 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:02.234020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:02.233988 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:04.489099 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:04.489064 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:06.593813 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:56:06.593774 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:56:09.506137 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.506103 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm"] Oct 31 03:56:09.510221 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.510195 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.510367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.510214 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:09.573124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.571989 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h"] Oct 31 03:56:09.580125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.580077 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.612803 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.612771 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm"] Oct 31 03:56:09.648926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.648890 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.648926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.648933 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.648957 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649197 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649129 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.649286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649194 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649237 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649278 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.649500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649423 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649471 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.649596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649571 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649652 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649684 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwst7\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.649824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649808 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.649883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649838 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fqb\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.649970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649950 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.650015 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.649998 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.650068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.650054 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.650145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.650128 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.665617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.665584 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h"] Oct 31 03:56:09.751285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751247 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.751532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751296 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751327 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751363 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.751532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751389 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751430 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751532 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751471 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751564 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751608 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751645 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751728 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751752 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwst7\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751752 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751882 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.751983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751921 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fqb\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.752428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752003 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.752428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752047 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.752428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752112 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.752428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752146 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.752642 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752444 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.752722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752699 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.752820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752799 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.753793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.752942 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.753793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.753223 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.753793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.751752 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.753793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.753474 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.753793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.753553 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.756633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.756537 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.758676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.758642 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.758953 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.758934 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.760859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.760718 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.761072 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.761042 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.813891 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.813855 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwst7\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.814066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.813855 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fqb\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.814435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.814409 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:09.814635 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.814617 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.824534 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.824512 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:09.905877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:09.905628 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:10.256944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.256909 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm"] Oct 31 03:56:10.258061 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:56:10.258034 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5826cb5a_f4d1_4fcb_84aa_eda4866fcff0.slice/crio-e9159926c60f979854a4db1e75073be3a45757718035a048f6ad0ed696b7b352 WatchSource:0}: Error finding container e9159926c60f979854a4db1e75073be3a45757718035a048f6ad0ed696b7b352: Status 404 returned error can't find the container with id e9159926c60f979854a4db1e75073be3a45757718035a048f6ad0ed696b7b352 Oct 31 03:56:10.260861 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.260822 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:56:10.260952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.260906 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:56:10.260952 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.260936 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:56:10.316776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.316751 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h"] Oct 31 03:56:10.332606 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:56:10.332577 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9b9f45_bc13_4fcf_9c0d_3517560e10dc.slice/crio-8c4efd9dfd8f66506c8ec757f7a12c3d939dffe805875ebbbf6874ad73a180a2 WatchSource:0}: Error finding container 8c4efd9dfd8f66506c8ec757f7a12c3d939dffe805875ebbbf6874ad73a180a2: Status 404 returned error can't find the container with id 8c4efd9dfd8f66506c8ec757f7a12c3d939dffe805875ebbbf6874ad73a180a2 Oct 31 03:56:10.335556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.335523 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:56:10.335762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.335588 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:56:10.335762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.335619 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 03:56:10.773524 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.773473 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" event={"ID":"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0","Type":"ContainerStarted","Data":"78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0"} Oct 31 03:56:10.773995 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.773548 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" event={"ID":"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0","Type":"ContainerStarted","Data":"e9159926c60f979854a4db1e75073be3a45757718035a048f6ad0ed696b7b352"} Oct 31 03:56:10.777475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.777435 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" event={"ID":"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc","Type":"ContainerStarted","Data":"b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c"} Oct 31 03:56:10.777686 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.777493 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" event={"ID":"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc","Type":"ContainerStarted","Data":"8c4efd9dfd8f66506c8ec757f7a12c3d939dffe805875ebbbf6874ad73a180a2"} Oct 31 03:56:10.824992 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.824964 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:10.896868 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.896810 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podStartSLOduration=1.896793967 podStartE2EDuration="1.896793967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:10.894787626 +0000 UTC m=+2198.691505885" watchObservedRunningTime="2026-10-31 03:56:10.896793967 +0000 UTC m=+2198.693512227" Oct 31 03:56:10.906810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:10.906781 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:11.021018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:11.020950 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podStartSLOduration=2.020932619 podStartE2EDuration="2.020932619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:11.014829178 +0000 UTC m=+2198.811547448" watchObservedRunningTime="2026-10-31 03:56:11.020932619 +0000 UTC m=+2198.817650878" Oct 31 03:56:11.837312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:11.837262 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 03:56:11.837983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:11.837943 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 03:56:11.936969 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:11.936926 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 03:56:11.937346 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:11.937307 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 03:56:12.826125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:12.825600 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 03:56:12.826125 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:12.825669 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 03:56:12.914648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:12.914500 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 03:56:12.914648 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:12.914582 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 03:56:13.845118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:13.838548 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 03:56:13.845118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:13.838624 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 03:56:13.924181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:13.919518 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 03:56:13.924181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:13.919583 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 03:56:14.691670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:14.691640 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:14.841121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:14.829499 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 03:56:14.841121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:14.829557 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 03:56:14.912116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:14.908474 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 03:56:14.912116 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:14.908538 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 03:56:15.830117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:15.829512 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 03:56:15.830117 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:15.829575 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 03:56:15.911132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:15.910439 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 03:56:15.911132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:15.910513 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 03:56:16.825619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:16.825578 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 03:56:16.825904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:16.825645 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 03:56:16.907037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:16.906989 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 31 03:56:16.907609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:16.907057 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 31 03:56:17.830266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:17.830235 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:17.830617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:17.830592 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:17.831442 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:17.831421 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:56:17.911740 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:17.911710 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:17.912316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:17.912113 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:17.913028 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:17.913010 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:56:19.560733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:19.560695 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8"] Oct 31 03:56:19.646724 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:19.646672 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:19.692678 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:19.691147 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv"] Oct 31 03:56:19.942053 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:19.942020 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:20.098762 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:20.098717 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:24.529339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.529304 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:24.561935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.561861 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" podUID="cb932f75-50e8-43e1-b2d7-7d2b72e56146" containerName="payload-processing" containerID="cri-o://3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1" gracePeriod=25 Oct 31 03:56:24.692260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.692214 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" podUID="fbd52fc4-6175-4651-866f-c2afaec5c206" containerName="payload-pre-processing" containerID="cri-o://473f0aebba9c127a172154e24d25ea18bab566693b6c1c668e906c01d10b315d" gracePeriod=25 Oct 31 03:56:24.892367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.892339 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:56:24.916033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.915992 4488 generic.go:356] "Generic (PLEG): container finished" podID="fbd52fc4-6175-4651-866f-c2afaec5c206" containerID="473f0aebba9c127a172154e24d25ea18bab566693b6c1c668e906c01d10b315d" exitCode=0 Oct 31 03:56:24.916250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.916067 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" event={"ID":"fbd52fc4-6175-4651-866f-c2afaec5c206","Type":"ContainerDied","Data":"473f0aebba9c127a172154e24d25ea18bab566693b6c1c668e906c01d10b315d"} Oct 31 03:56:24.918155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.918130 4488 generic.go:356] "Generic (PLEG): container finished" podID="cb932f75-50e8-43e1-b2d7-7d2b72e56146" containerID="3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1" exitCode=0 Oct 31 03:56:24.918308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.918247 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" event={"ID":"cb932f75-50e8-43e1-b2d7-7d2b72e56146","Type":"ContainerDied","Data":"3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1"} Oct 31 03:56:24.918308 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.918303 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" event={"ID":"cb932f75-50e8-43e1-b2d7-7d2b72e56146","Type":"ContainerDied","Data":"a621f980a44e0d3a49e82e14f2f5fd1c98d8d344f9c718f1771fc4d1ef7667ba"} Oct 31 03:56:24.918424 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.918332 4488 scope.go:122] "RemoveContainer" containerID="3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1" Oct 31 03:56:24.918538 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.918522 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8" Oct 31 03:56:24.930869 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.930843 4488 scope.go:122] "RemoveContainer" containerID="3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1" Oct 31 03:56:24.931313 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:56:24.931274 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1\": container with ID starting with 3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1 not found: ID does not exist" containerID="3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1" Oct 31 03:56:24.931433 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.931316 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1"} err="failed to get container status \"3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1\": rpc error: code = NotFound desc = could not find container \"3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1\": container with ID starting with 3d35f8ab36c660e6a99ada1e5b3534d19775083f9328196dc293a7b30ea086a1 not found: ID does not exist" Oct 31 03:56:24.992823 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:24.992798 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:56:25.047397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.047350 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg\" (UniqueName: \"kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg\") pod \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " Oct 31 03:56:25.047681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.047664 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume\") pod \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\" (UID: \"cb932f75-50e8-43e1-b2d7-7d2b72e56146\") " Oct 31 03:56:25.048163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.048132 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume" pod "cb932f75-50e8-43e1-b2d7-7d2b72e56146" (UID: "cb932f75-50e8-43e1-b2d7-7d2b72e56146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:25.048332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.048312 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb932f75-50e8-43e1-b2d7-7d2b72e56146-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.049849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.049819 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg" pod "cb932f75-50e8-43e1-b2d7-7d2b72e56146" (UID: "cb932f75-50e8-43e1-b2d7-7d2b72e56146"). InnerVolumeSpecName "kube-api-access-48tfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:25.149784 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.149736 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc\" (UniqueName: \"kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc\") pod \"fbd52fc4-6175-4651-866f-c2afaec5c206\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " Oct 31 03:56:25.149979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.149847 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume\") pod \"fbd52fc4-6175-4651-866f-c2afaec5c206\" (UID: \"fbd52fc4-6175-4651-866f-c2afaec5c206\") " Oct 31 03:56:25.150258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.150234 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume" pod "fbd52fc4-6175-4651-866f-c2afaec5c206" (UID: "fbd52fc4-6175-4651-866f-c2afaec5c206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:25.150574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.150528 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd52fc4-6175-4651-866f-c2afaec5c206-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.150574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.150546 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48tfg\" (UniqueName: \"kubernetes.io/projected/cb932f75-50e8-43e1-b2d7-7d2b72e56146-kube-api-access-48tfg\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.152251 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.152204 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc" pod "fbd52fc4-6175-4651-866f-c2afaec5c206" (UID: "fbd52fc4-6175-4651-866f-c2afaec5c206"). InnerVolumeSpecName "kube-api-access-9jtjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:25.252465 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.252418 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jtjc\" (UniqueName: \"kubernetes.io/projected/fbd52fc4-6175-4651-866f-c2afaec5c206-kube-api-access-9jtjc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.343281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.343234 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8"] Oct 31 03:56:25.357705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.357667 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-36664b-5458ddfc47-878h8"] Oct 31 03:56:25.927041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.926939 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" Oct 31 03:56:25.927041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.926959 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv" event={"ID":"fbd52fc4-6175-4651-866f-c2afaec5c206","Type":"ContainerDied","Data":"ed36f0597e7bc2a28a50a5a5cdc4a73c72b924db8e7993a8aca1648c0ecc085b"} Oct 31 03:56:25.927659 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:25.927053 4488 scope.go:122] "RemoveContainer" containerID="473f0aebba9c127a172154e24d25ea18bab566693b6c1c668e906c01d10b315d" Oct 31 03:56:26.092397 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:26.092357 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv"] Oct 31 03:56:26.142342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:26.142308 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-36664b-5c59dc9979-zdkwv"] Oct 31 03:56:26.317710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:26.317631 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb932f75-50e8-43e1-b2d7-7d2b72e56146" path="/var/lib/kubelet/pods/cb932f75-50e8-43e1-b2d7-7d2b72e56146/volumes" Oct 31 03:56:26.318157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:26.318141 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd52fc4-6175-4651-866f-c2afaec5c206" path="/var/lib/kubelet/pods/fbd52fc4-6175-4651-866f-c2afaec5c206/volumes" Oct 31 03:56:29.358893 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:29.358859 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:29.652038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:29.651927 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:34.180373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:34.180333 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:35.217888 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:35.217844 4488 scope.go:122] "RemoveContainer" containerID="d3dec56b83b8907750984beb799d3a517c9a75d6b83d954b6b7b50948b5d42b7" Oct 31 03:56:35.227852 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:35.227823 4488 scope.go:122] "RemoveContainer" containerID="8f453e12ab9211d2817fbe474140527a96847fe288f1bbbb7c2019dbbc71a58b" Oct 31 03:56:35.239436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:35.239412 4488 scope.go:122] "RemoveContainer" containerID="24d5a7f1f8ff7787ca22821974931baedd2b506d2a0fef792970dd343a7424ac" Oct 31 03:56:36.590322 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:56:36.590291 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:56:38.480262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:38.480223 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:38.489351 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:38.489319 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6"] Oct 31 03:56:38.493564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:38.493509 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" podUID="6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" containerName="maas-api" containerID="cri-o://e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6" gracePeriod=30 Oct 31 03:56:39.008965 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.008590 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:56:39.021507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.020499 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs\") pod \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " Oct 31 03:56:39.021507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.020570 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\") pod \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " Oct 31 03:56:39.021507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.020713 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca\") pod \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " Oct 31 03:56:39.021507 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.020823 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr\" (UniqueName: \"kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr\") pod \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\" (UID: \"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9\") " Oct 31 03:56:39.022639 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.022348 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca" pod "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" (UID: "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:39.025420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.025373 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs" pod "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" (UID: "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:39.028689 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.028643 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr" pod "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" (UID: "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9"). InnerVolumeSpecName "kube-api-access-wsbsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:39.031011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.030948 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls" pod "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" (UID: "6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:39.038790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.038760 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" containerID="e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6" exitCode=0 Oct 31 03:56:39.039005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.038829 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" Oct 31 03:56:39.039219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.038852 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" event={"ID":"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9","Type":"ContainerDied","Data":"e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6"} Oct 31 03:56:39.039314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.039264 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6" event={"ID":"6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9","Type":"ContainerDied","Data":"143d1cbeb07e36a2ab3b197d387a0f7165bd836e07ad80047fa455f6dfaddde9"} Oct 31 03:56:39.039314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.039306 4488 scope.go:122] "RemoveContainer" containerID="e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6" Oct 31 03:56:39.064742 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.064714 4488 scope.go:122] "RemoveContainer" containerID="e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6" Oct 31 03:56:39.065243 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:56:39.065209 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6\": container with ID starting with e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6 not found: ID does not exist" containerID="e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6" Oct 31 03:56:39.065367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.065255 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6"} err="failed to get container status \"e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6\": rpc error: code = NotFound desc = could not find container \"e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6\": container with ID starting with e1c2f4b14b9f82ae5debf3b6d1354c3a51a68b92905eb7dcc607cdfd7b0ad4e6 not found: ID does not exist" Oct 31 03:56:39.123314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.123200 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:39.123314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.123241 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:39.123314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.123263 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:39.123314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.123280 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbsr\" (UniqueName: \"kubernetes.io/projected/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9-kube-api-access-wsbsr\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:39.243287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.243193 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6"] Oct 31 03:56:39.249328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:39.249293 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bc91a3-676db8c4dd-jnlp6"] Oct 31 03:56:40.119848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:40.119814 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:40.317629 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:40.317599 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" path="/var/lib/kubelet/pods/6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9/volumes" Oct 31 03:56:40.459788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:40.459712 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:40.614188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:40.614156 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:41.223078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:41.223044 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:45.679841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:45.679810 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:47.826181 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:47.826139 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:48.072799 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:48.072765 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:48.228076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:48.228037 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs"] Oct 31 03:56:48.228448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:48.228374 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" containerID="cri-o://f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27" gracePeriod=30 Oct 31 03:56:48.236250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:48.236218 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:50.612328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:50.612294 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:51.005881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:51.005798 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:52.749588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:52.749548 4488 patch_prober.go:29] interesting pod/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:52.749997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:52.749611 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:53.514480 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.514455 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:56:53.655502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655470 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.655705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655518 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.655705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655540 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.655705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655565 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.655705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655639 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.655705 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655681 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.656014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655783 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.656014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655794 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:53.656014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655891 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.656014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655940 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf\") pod \"8114c66e-0643-42d4-9ee0-e9c9d1923852\" (UID: \"8114c66e-0643-42d4-9ee0-e9c9d1923852\") " Oct 31 03:56:53.656014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.655928 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:53.656336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656205 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:53.656336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656213 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:53.656336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656296 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:53.656790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656766 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.656940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656792 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.656940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656807 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.656940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656823 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.656940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.656837 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8114c66e-0643-42d4-9ee0-e9c9d1923852-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.658741 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.658708 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:53.658844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.658789 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:53.658890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.658868 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "kube-api-access-75hwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:53.658990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.658972 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token" pod "8114c66e-0643-42d4-9ee0-e9c9d1923852" (UID: "8114c66e-0643-42d4-9ee0-e9c9d1923852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:53.757963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.757874 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.757963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.757905 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.757963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.757915 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:53.757963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:53.757925 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75hwf\" (UniqueName: \"kubernetes.io/projected/8114c66e-0643-42d4-9ee0-e9c9d1923852-kube-api-access-75hwf\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:56:54.143945 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.143904 4488 generic.go:356] "Generic (PLEG): container finished" podID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerID="f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27" exitCode=0 Oct 31 03:56:54.144203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.143981 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" Oct 31 03:56:54.144203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.143996 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" event={"ID":"8114c66e-0643-42d4-9ee0-e9c9d1923852","Type":"ContainerDied","Data":"f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27"} Oct 31 03:56:54.144203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.144067 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs" event={"ID":"8114c66e-0643-42d4-9ee0-e9c9d1923852","Type":"ContainerDied","Data":"ae2d2deed260019cc8ad10646643c754cdedaf8ec097fcb1b6402ea0b8addcbb"} Oct 31 03:56:54.144203 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.144113 4488 scope.go:122] "RemoveContainer" containerID="f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27" Oct 31 03:56:54.155455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.155435 4488 scope.go:122] "RemoveContainer" containerID="57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2" Oct 31 03:56:54.164818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.164799 4488 scope.go:122] "RemoveContainer" containerID="f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27" Oct 31 03:56:54.165100 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:56:54.165066 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27\": container with ID starting with f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27 not found: ID does not exist" containerID="f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27" Oct 31 03:56:54.165162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.165137 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27"} err="failed to get container status \"f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27\": rpc error: code = NotFound desc = could not find container \"f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27\": container with ID starting with f219469eae801bad97c5531940a0f6b561a2a6ceab7112ef58b745a7336d0a27 not found: ID does not exist" Oct 31 03:56:54.165162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.165157 4488 scope.go:122] "RemoveContainer" containerID="57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2" Oct 31 03:56:54.165442 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:56:54.165421 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2\": container with ID starting with 57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2 not found: ID does not exist" containerID="57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2" Oct 31 03:56:54.165553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.165447 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2"} err="failed to get container status \"57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2\": rpc error: code = NotFound desc = could not find container \"57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2\": container with ID starting with 57ef2b5394a97cf99915d947c46d2a0cc3cd9d95969ef694339ae6aa4e4f2af2 not found: ID does not exist" Oct 31 03:56:54.318637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.318609 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs"] Oct 31 03:56:54.363453 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.363415 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bc91a3-openshift-default-5cf5ddbcff-6hsxs"] Oct 31 03:56:54.846722 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:54.846689 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.184451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:56.184414 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.317598 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:56.317567 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" path="/var/lib/kubelet/pods/8114c66e-0643-42d4-9ee0-e9c9d1923852/volumes" Oct 31 03:56:58.624260 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:58.624225 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c"] Oct 31 03:56:58.624700 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:56:58.624452 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" containerID="cri-o://5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b" gracePeriod=30 Oct 31 03:57:00.886178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:00.886146 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:01.041718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:01.041675 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:01.363570 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:01.363530 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:01.909730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:01.909692 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn"] Oct 31 03:57:01.910234 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:01.909938 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" containerID="cri-o://403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0" gracePeriod=30 Oct 31 03:57:03.379547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:03.379514 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:03.605047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:03.605007 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:03.892830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:03.892761 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:57:04.034312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034279 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034348 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034478 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034531 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034512 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034550 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034579 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034635 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034670 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.034913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034787 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:04.034913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.034813 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket\") pod \"228357ef-0268-409f-bd42-73f83b0567de\" (UID: \"228357ef-0268-409f-bd42-73f83b0567de\") " Oct 31 03:57:04.035212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035171 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:04.035330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035157 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:04.035425 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035397 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:04.035576 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035419 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:04.035653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035615 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.035653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035629 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.035653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035640 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.035653 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035648 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.035858 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.035660 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228357ef-0268-409f-bd42-73f83b0567de-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.036913 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.036890 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:04.037020 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.036941 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:04.037256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.037240 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:04.037435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.037420 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q" pod "228357ef-0268-409f-bd42-73f83b0567de" (UID: "228357ef-0268-409f-bd42-73f83b0567de"). InnerVolumeSpecName "kube-api-access-kkq7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:04.137194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.137151 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.137194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.137182 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228357ef-0268-409f-bd42-73f83b0567de-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.137194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.137193 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228357ef-0268-409f-bd42-73f83b0567de-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.137194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.137203 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkq7q\" (UniqueName: \"kubernetes.io/projected/228357ef-0268-409f-bd42-73f83b0567de-kube-api-access-kkq7q\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:04.258239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.258151 4488 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:04.258239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.258210 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:04.263636 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.263605 4488 generic.go:356] "Generic (PLEG): container finished" podID="228357ef-0268-409f-bd42-73f83b0567de" containerID="5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b" exitCode=0 Oct 31 03:57:04.263794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.263668 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" Oct 31 03:57:04.263794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.263683 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" event={"ID":"228357ef-0268-409f-bd42-73f83b0567de","Type":"ContainerDied","Data":"5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b"} Oct 31 03:57:04.263794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.263747 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c" event={"ID":"228357ef-0268-409f-bd42-73f83b0567de","Type":"ContainerDied","Data":"67d50f6ab0c3a2141af7a58ffba6a05d8cc7abfc9b833afeae4ad618829280db"} Oct 31 03:57:04.263794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.263773 4488 scope.go:122] "RemoveContainer" containerID="5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b" Oct 31 03:57:04.274971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.274952 4488 scope.go:122] "RemoveContainer" containerID="97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6" Oct 31 03:57:04.284745 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.284721 4488 scope.go:122] "RemoveContainer" containerID="5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b" Oct 31 03:57:04.285047 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:04.285028 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b\": container with ID starting with 5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b not found: ID does not exist" containerID="5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b" Oct 31 03:57:04.285174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.285061 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b"} err="failed to get container status \"5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b\": rpc error: code = NotFound desc = could not find container \"5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b\": container with ID starting with 5187537f9acc8696627a2ec08216c11b8534ba96cbd3017a18595e75dfc7152b not found: ID does not exist" Oct 31 03:57:04.285252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.285215 4488 scope.go:122] "RemoveContainer" containerID="97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6" Oct 31 03:57:04.285509 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:04.285490 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6\": container with ID starting with 97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6 not found: ID does not exist" containerID="97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6" Oct 31 03:57:04.285559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.285515 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6"} err="failed to get container status \"97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6\": rpc error: code = NotFound desc = could not find container \"97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6\": container with ID starting with 97beeac83d03f01deaaabdd180ed09f0739197661a1f7eb37655a946e5689da6 not found: ID does not exist" Oct 31 03:57:04.432083 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.432046 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c"] Oct 31 03:57:04.436076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:04.436047 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7c43a6-openshift-default-7685bfcd7c-6mc9c"] Oct 31 03:57:05.116212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:05.116164 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:57:06.011855 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:06.011796 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:06.359437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:06.359403 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228357ef-0268-409f-bd42-73f83b0567de" path="/var/lib/kubelet/pods/228357ef-0268-409f-bd42-73f83b0567de/volumes" Oct 31 03:57:06.596256 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:06.596215 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:57:07.176626 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.176592 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:57:07.309863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309767 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.309863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309803 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.309863 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309848 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309910 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309936 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309963 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.309994 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310059 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310267 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:07.310462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310284 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:07.310462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310295 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:07.310462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310276 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data\") pod \"b64704aa-9fdd-4ac0-843c-7f699375d126\" (UID: \"b64704aa-9fdd-4ac0-843c-7f699375d126\") " Oct 31 03:57:07.310462 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310192 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:07.310886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310463 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:07.311017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.310945 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.311017 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.311020 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.311258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.311038 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64704aa-9fdd-4ac0-843c-7f699375d126-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.311258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.311053 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.311258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.311067 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.312490 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.312453 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:07.312612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.312567 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "kube-api-access-glmkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:07.312612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.312567 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:07.312612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.312599 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo" pod "b64704aa-9fdd-4ac0-843c-7f699375d126" (UID: "b64704aa-9fdd-4ac0-843c-7f699375d126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:07.365735 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.365695 4488 generic.go:356] "Generic (PLEG): container finished" podID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerID="403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0" exitCode=0 Oct 31 03:57:07.365908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.365742 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" Oct 31 03:57:07.365908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.365773 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" event={"ID":"b64704aa-9fdd-4ac0-843c-7f699375d126","Type":"ContainerDied","Data":"403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0"} Oct 31 03:57:07.365908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.365832 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn" event={"ID":"b64704aa-9fdd-4ac0-843c-7f699375d126","Type":"ContainerDied","Data":"0c15b97c4d51e2114643ec7e4f5d45f05f4b290b70b5d7d2dbe2ea0791500428"} Oct 31 03:57:07.365908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.365871 4488 scope.go:122] "RemoveContainer" containerID="403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0" Oct 31 03:57:07.377865 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.377839 4488 scope.go:122] "RemoveContainer" containerID="403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0" Oct 31 03:57:07.378169 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:07.378150 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0\": container with ID starting with 403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0 not found: ID does not exist" containerID="403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0" Oct 31 03:57:07.378283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.378178 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0"} err="failed to get container status \"403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0\": rpc error: code = NotFound desc = could not find container \"403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0\": container with ID starting with 403571b5b3074e5475ba254771781dfeb5bb8dd3b247174aa00ca40456a5e8a0 not found: ID does not exist" Oct 31 03:57:07.412381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.412347 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.412381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.412377 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.412381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.412387 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glmkv\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-kube-api-access-glmkv\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.412381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.412397 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64704aa-9fdd-4ac0-843c-7f699375d126-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:07.532463 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.532424 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn"] Oct 31 03:57:07.537468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.537439 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-z9qqn"] Oct 31 03:57:07.582523 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.582494 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:07.916258 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:07.916167 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:08.073367 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:08.073327 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:08.316867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:08.316767 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" path="/var/lib/kubelet/pods/b64704aa-9fdd-4ac0-843c-7f699375d126/volumes" Oct 31 03:57:08.563592 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:08.563549 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:13.000825 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:13.000789 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:14.507723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:14.507684 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:14.664667 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:14.664634 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:14.877064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:14.877034 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:17.995571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:17.995528 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:22.918711 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:22.918674 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:27.078931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.078889 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx"] Oct 31 03:57:27.079395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.079168 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" podUID="4193e587-c7b7-497e-95c5-3327ac467cfd" containerName="maas-api" containerID="cri-o://22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765" gracePeriod=30 Oct 31 03:57:27.192926 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.192886 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:27.405330 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.405306 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:57:27.525488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.525451 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs\") pod \"4193e587-c7b7-497e-95c5-3327ac467cfd\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " Oct 31 03:57:27.525788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.525577 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca\") pod \"4193e587-c7b7-497e-95c5-3327ac467cfd\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " Oct 31 03:57:27.525788 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.525635 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc\" (UniqueName: \"kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc\") pod \"4193e587-c7b7-497e-95c5-3327ac467cfd\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " Oct 31 03:57:27.525923 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.525812 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\") pod \"4193e587-c7b7-497e-95c5-3327ac467cfd\" (UID: \"4193e587-c7b7-497e-95c5-3327ac467cfd\") " Oct 31 03:57:27.527410 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.527319 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca" pod "4193e587-c7b7-497e-95c5-3327ac467cfd" (UID: "4193e587-c7b7-497e-95c5-3327ac467cfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:27.550359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.544233 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs" pod "4193e587-c7b7-497e-95c5-3327ac467cfd" (UID: "4193e587-c7b7-497e-95c5-3327ac467cfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:27.557141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.554734 4488 generic.go:356] "Generic (PLEG): container finished" podID="4193e587-c7b7-497e-95c5-3327ac467cfd" containerID="22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765" exitCode=0 Oct 31 03:57:27.557141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.554812 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" event={"ID":"4193e587-c7b7-497e-95c5-3327ac467cfd","Type":"ContainerDied","Data":"22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765"} Oct 31 03:57:27.557141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.554863 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" event={"ID":"4193e587-c7b7-497e-95c5-3327ac467cfd","Type":"ContainerDied","Data":"f8f701e98f594b2123ac20cbf8f863ef4bde83b1e286341aa253bb0d84cb4539"} Oct 31 03:57:27.557141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.554897 4488 scope.go:122] "RemoveContainer" containerID="22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765" Oct 31 03:57:27.557141 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.555108 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx" Oct 31 03:57:27.558359 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.558287 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls" pod "4193e587-c7b7-497e-95c5-3327ac467cfd" (UID: "4193e587-c7b7-497e-95c5-3327ac467cfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:27.564439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.564389 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc" pod "4193e587-c7b7-497e-95c5-3327ac467cfd" (UID: "4193e587-c7b7-497e-95c5-3327ac467cfd"). InnerVolumeSpecName "kube-api-access-nzsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:27.578959 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.578738 4488 scope.go:122] "RemoveContainer" containerID="22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765" Oct 31 03:57:27.579504 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:27.579475 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765\": container with ID starting with 22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765 not found: ID does not exist" containerID="22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765" Oct 31 03:57:27.579615 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.579518 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765"} err="failed to get container status \"22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765\": rpc error: code = NotFound desc = could not find container \"22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765\": container with ID starting with 22b63144329cc5779e8e718fbff30bcd66808a2742da3b57c82de2761a3ea765 not found: ID does not exist" Oct 31 03:57:27.587621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.587586 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:27.628357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.628245 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193e587-c7b7-497e-95c5-3327ac467cfd-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.628357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.628279 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzsnc\" (UniqueName: \"kubernetes.io/projected/4193e587-c7b7-497e-95c5-3327ac467cfd-kube-api-access-nzsnc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.628357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.628296 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.628357 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.628310 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4193e587-c7b7-497e-95c5-3327ac467cfd-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.785603 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.785568 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:27.977336 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.977238 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx"] Oct 31 03:57:27.980877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:27.980847 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-36664b-76b5bd74bf-5hctx"] Oct 31 03:57:28.317901 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:28.317822 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4193e587-c7b7-497e-95c5-3327ac467cfd" path="/var/lib/kubelet/pods/4193e587-c7b7-497e-95c5-3327ac467cfd/volumes" Oct 31 03:57:28.750961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:28.750924 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:28.960844 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:28.960813 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:29.141289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:29.130293 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:29.681103 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:29.681055 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:30.037881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:30.037786 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:34.012347 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:34.012312 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:35.757671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:35.757614 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:35.912174 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:35.912135 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:36.092445 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:36.092411 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p"] Oct 31 03:57:36.092679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:36.092646 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" containerID="cri-o://130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb" gracePeriod=30 Oct 31 03:57:36.112504 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:36.112473 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:36.591080 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:36.591033 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:57:39.312550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:39.312511 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:41.256620 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.256575 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl"] Oct 31 03:57:41.257105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.256861 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" containerID="cri-o://65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f" gracePeriod=30 Oct 31 03:57:41.370720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.370631 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:57:41.382643 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382612 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.382836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382672 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.382836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382703 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.382836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382782 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.382836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382807 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.383062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382867 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.383062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382924 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.383062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382968 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.383062 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.382999 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket\") pod \"4d7b08bd-1c06-4308-84ed-f71ef4043463\" (UID: \"4d7b08bd-1c06-4308-84ed-f71ef4043463\") " Oct 31 03:57:41.383314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383067 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:41.383314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383076 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:41.383314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383302 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:41.383645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383384 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:41.383768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383746 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:41.383981 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383941 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7b08bd-1c06-4308-84ed-f71ef4043463-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.384071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.383986 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.384071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.384004 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.384071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.384018 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.384071 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.384032 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.385294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.385268 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:41.385294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.385274 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:41.385419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.385360 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "kube-api-access-g9h96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:41.385468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.385455 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy" pod "4d7b08bd-1c06-4308-84ed-f71ef4043463" (UID: "4d7b08bd-1c06-4308-84ed-f71ef4043463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:41.485066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.485023 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9h96\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-kube-api-access-g9h96\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.485066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.485058 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.485066 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.485071 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.485343 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.485083 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7b08bd-1c06-4308-84ed-f71ef4043463-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:41.654551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.654461 4488 generic.go:356] "Generic (PLEG): container finished" podID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerID="130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb" exitCode=0 Oct 31 03:57:41.654551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.654538 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" Oct 31 03:57:41.654765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.654546 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" event={"ID":"4d7b08bd-1c06-4308-84ed-f71ef4043463","Type":"ContainerDied","Data":"130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb"} Oct 31 03:57:41.654765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.654608 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p" event={"ID":"4d7b08bd-1c06-4308-84ed-f71ef4043463","Type":"ContainerDied","Data":"604b5d5ef931ef7f373e82e81e40f60534fb86a7309e80f7bd97906957f178c7"} Oct 31 03:57:41.654765 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.654638 4488 scope.go:122] "RemoveContainer" containerID="130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb" Oct 31 03:57:41.667153 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.667133 4488 scope.go:122] "RemoveContainer" containerID="661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1" Oct 31 03:57:41.677068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.677046 4488 scope.go:122] "RemoveContainer" containerID="130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb" Oct 31 03:57:41.677418 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:41.677398 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb\": container with ID starting with 130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb not found: ID does not exist" containerID="130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb" Oct 31 03:57:41.677474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.677428 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb"} err="failed to get container status \"130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb\": rpc error: code = NotFound desc = could not find container \"130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb\": container with ID starting with 130fd0ded400064b6146ae952b23734c995c6e25a97c8eb7138a89cb021fecbb not found: ID does not exist" Oct 31 03:57:41.677474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.677448 4488 scope.go:122] "RemoveContainer" containerID="661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1" Oct 31 03:57:41.677705 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:41.677688 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1\": container with ID starting with 661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1 not found: ID does not exist" containerID="661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1" Oct 31 03:57:41.677761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.677711 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1"} err="failed to get container status \"661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1\": rpc error: code = NotFound desc = could not find container \"661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1\": container with ID starting with 661593c32f2b67193d3a9fdb0fed9d6f9fa6d9a04c08ce25aea6d8c584b1b3d1 not found: ID does not exist" Oct 31 03:57:41.821702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.821654 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p"] Oct 31 03:57:41.826997 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:41.826961 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-36664b-openshift-default-7bd9b7b76f-ngw8p"] Oct 31 03:57:42.317227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:42.317198 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" path="/var/lib/kubelet/pods/4d7b08bd-1c06-4308-84ed-f71ef4043463/volumes" Oct 31 03:57:43.691245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:43.691213 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc"] Oct 31 03:57:43.691723 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:43.691403 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" podUID="0fee47dc-7209-46db-8f3c-a27b016a114f" containerName="maas-api" containerID="cri-o://deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90" gracePeriod=30 Oct 31 03:57:43.802670 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:43.802629 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:44.013657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.013632 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:57:44.128052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.125607 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l\" (UniqueName: \"kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l\") pod \"0fee47dc-7209-46db-8f3c-a27b016a114f\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " Oct 31 03:57:44.128052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.125761 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\") pod \"0fee47dc-7209-46db-8f3c-a27b016a114f\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " Oct 31 03:57:44.128052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.125927 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs\") pod \"0fee47dc-7209-46db-8f3c-a27b016a114f\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " Oct 31 03:57:44.128052 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.126019 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca\") pod \"0fee47dc-7209-46db-8f3c-a27b016a114f\" (UID: \"0fee47dc-7209-46db-8f3c-a27b016a114f\") " Oct 31 03:57:44.129132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.128646 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca" pod "0fee47dc-7209-46db-8f3c-a27b016a114f" (UID: "0fee47dc-7209-46db-8f3c-a27b016a114f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:44.133633 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.133359 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs" pod "0fee47dc-7209-46db-8f3c-a27b016a114f" (UID: "0fee47dc-7209-46db-8f3c-a27b016a114f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:44.136677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.136648 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls" pod "0fee47dc-7209-46db-8f3c-a27b016a114f" (UID: "0fee47dc-7209-46db-8f3c-a27b016a114f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:44.138147 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.138045 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l" pod "0fee47dc-7209-46db-8f3c-a27b016a114f" (UID: "0fee47dc-7209-46db-8f3c-a27b016a114f"). InnerVolumeSpecName "kube-api-access-qt58l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:44.228591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.228549 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:44.228591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.228585 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee47dc-7209-46db-8f3c-a27b016a114f-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:44.228591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.228594 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt58l\" (UniqueName: \"kubernetes.io/projected/0fee47dc-7209-46db-8f3c-a27b016a114f-kube-api-access-qt58l\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:44.228854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.228604 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fee47dc-7209-46db-8f3c-a27b016a114f-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:44.675885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.675851 4488 generic.go:356] "Generic (PLEG): container finished" podID="0fee47dc-7209-46db-8f3c-a27b016a114f" containerID="deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90" exitCode=0 Oct 31 03:57:44.676145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.675904 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" Oct 31 03:57:44.676145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.675921 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" event={"ID":"0fee47dc-7209-46db-8f3c-a27b016a114f","Type":"ContainerDied","Data":"deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90"} Oct 31 03:57:44.676145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.676003 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc" event={"ID":"0fee47dc-7209-46db-8f3c-a27b016a114f","Type":"ContainerDied","Data":"90281fecdfab99e8ca3ff54fa05b215d5c46da852dc708e988c176861a157b9d"} Oct 31 03:57:44.676145 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.676035 4488 scope.go:122] "RemoveContainer" containerID="deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90" Oct 31 03:57:44.686212 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.686195 4488 scope.go:122] "RemoveContainer" containerID="deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90" Oct 31 03:57:44.686512 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:44.686494 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90\": container with ID starting with deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90 not found: ID does not exist" containerID="deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90" Oct 31 03:57:44.686560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.686520 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90"} err="failed to get container status \"deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90\": rpc error: code = NotFound desc = could not find container \"deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90\": container with ID starting with deea9092f488d537b13bd997207ebba5a637e0a09d92bb069f546b47db470a90 not found: ID does not exist" Oct 31 03:57:44.848199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.848157 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc"] Oct 31 03:57:44.895407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:44.895363 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1180d818-568bbbfc97-z8zxc"] Oct 31 03:57:45.143501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:45.143469 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:45.543106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:45.542913 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:45.823409 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:45.823271 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:46.318150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.318118 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fee47dc-7209-46db-8f3c-a27b016a114f" path="/var/lib/kubelet/pods/0fee47dc-7209-46db-8f3c-a27b016a114f/volumes" Oct 31 03:57:46.533446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.533410 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:46.561224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.561194 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:57:46.673599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673559 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673621 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673667 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673702 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673807 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673791 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673820 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673946 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.673988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.673982 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.674621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674122 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data\") pod \"07c5632c-d89d-4db0-a838-b5b40db09ee8\" (UID: \"07c5632c-d89d-4db0-a838-b5b40db09ee8\") " Oct 31 03:57:46.674621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674155 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:46.674621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674472 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:46.674621 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674504 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:46.674842 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674807 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.674842 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674830 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.674950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.674845 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.675476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.675415 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:46.675936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.675847 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:46.689126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.687891 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:46.689126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.687961 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:46.689126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.688019 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:46.691041 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.691008 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb" pod "07c5632c-d89d-4db0-a838-b5b40db09ee8" (UID: "07c5632c-d89d-4db0-a838-b5b40db09ee8"). InnerVolumeSpecName "kube-api-access-jhpmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:46.698337 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.698302 4488 generic.go:356] "Generic (PLEG): container finished" podID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerID="65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f" exitCode=0 Oct 31 03:57:46.698501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.698388 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" Oct 31 03:57:46.698501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.698389 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" event={"ID":"07c5632c-d89d-4db0-a838-b5b40db09ee8","Type":"ContainerDied","Data":"65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f"} Oct 31 03:57:46.698501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.698464 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl" event={"ID":"07c5632c-d89d-4db0-a838-b5b40db09ee8","Type":"ContainerDied","Data":"cf87752708120b7dcbd913d81b10bc12ca8589a076653ee6eac0f5ee66c9dda1"} Oct 31 03:57:46.698501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.698493 4488 scope.go:122] "RemoveContainer" containerID="65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f" Oct 31 03:57:46.712222 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.712184 4488 scope.go:122] "RemoveContainer" containerID="215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e" Oct 31 03:57:46.723824 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.723803 4488 scope.go:122] "RemoveContainer" containerID="65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f" Oct 31 03:57:46.724301 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:46.724268 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f\": container with ID starting with 65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f not found: ID does not exist" containerID="65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f" Oct 31 03:57:46.724415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.724315 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f"} err="failed to get container status \"65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f\": rpc error: code = NotFound desc = could not find container \"65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f\": container with ID starting with 65969bb7f2a553b5fa011f4ad45480f1a227473fa46c0549d448e716b26d047f not found: ID does not exist" Oct 31 03:57:46.724415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.724342 4488 scope.go:122] "RemoveContainer" containerID="215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e" Oct 31 03:57:46.724687 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:57:46.724660 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e\": container with ID starting with 215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e not found: ID does not exist" containerID="215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e" Oct 31 03:57:46.724743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.724701 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e"} err="failed to get container status \"215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e\": rpc error: code = NotFound desc = could not find container \"215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e\": container with ID starting with 215d686cf555cfb8e96761d98caef4aad887b2a930e267afae2773258451531e not found: ID does not exist" Oct 31 03:57:46.775890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.775854 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5632c-d89d-4db0-a838-b5b40db09ee8-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.775890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.775889 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.775890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.775905 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpmb\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-kube-api-access-jhpmb\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.776276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.775928 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.776276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.775942 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.776276 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.775974 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5632c-d89d-4db0-a838-b5b40db09ee8-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:46.852996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.852956 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl"] Oct 31 03:57:46.856897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.856869 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-cbrcl"] Oct 31 03:57:46.876430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:46.876395 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:47.135352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:47.135247 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:47.305178 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:47.305137 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:47.802627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:47.802594 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:47.965289 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:47.965251 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:48.316971 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:48.316936 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" path="/var/lib/kubelet/pods/07c5632c-d89d-4db0-a838-b5b40db09ee8/volumes" Oct 31 03:57:49.047584 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:49.047550 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm"] Oct 31 03:57:49.097045 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:49.097012 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b37031-wmzhm"] Oct 31 03:57:49.243935 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:49.243889 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:49.406501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:49.406468 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:50.317885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:50.317849 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59cd3c8-ef2c-4786-84aa-bf1bc2682396" path="/var/lib/kubelet/pods/f59cd3c8-ef2c-4786-84aa-bf1bc2682396/volumes" Oct 31 03:57:51.459494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:51.459451 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn"] Oct 31 03:57:51.554864 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:51.554830 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:52.391320 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:52.391284 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:54.231575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:54.231544 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:54.388192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:54.388158 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:54.591804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:54.591773 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:55.464851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.464817 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz"] Oct 31 03:57:55.465312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465298 4488 cpu_manager.go:414] "containerMap: removing container" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" Oct 31 03:57:55.465312 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465313 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465336 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465341 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465348 4488 cpu_manager.go:414] "containerMap: removing container" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465354 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465364 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465369 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465383 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4193e587-c7b7-497e-95c5-3327ac467cfd" containerName="maas-api" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465388 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4193e587-c7b7-497e-95c5-3327ac467cfd" containerName="maas-api" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465395 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" Oct 31 03:57:55.465406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465400 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465410 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" containerName="maas-api" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465416 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" containerName="maas-api" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465423 4488 cpu_manager.go:414] "containerMap: removing container" podUID="fbd52fc4-6175-4651-866f-c2afaec5c206" containerName="payload-pre-processing" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465427 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd52fc4-6175-4651-866f-c2afaec5c206" containerName="payload-pre-processing" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465434 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0fee47dc-7209-46db-8f3c-a27b016a114f" containerName="maas-api" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465438 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fee47dc-7209-46db-8f3c-a27b016a114f" containerName="maas-api" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465450 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465454 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465460 4488 cpu_manager.go:414] "containerMap: removing container" podUID="cb932f75-50e8-43e1-b2d7-7d2b72e56146" containerName="payload-processing" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465464 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb932f75-50e8-43e1-b2d7-7d2b72e56146" containerName="payload-processing" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465564 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465573 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64704aa-9fdd-4ac0-843c-7f699375d126" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465581 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb932f75-50e8-43e1-b2d7-7d2b72e56146" containerName="payload-processing" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465587 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465595 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465600 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465605 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465612 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4193e587-c7b7-497e-95c5-3327ac467cfd" containerName="maas-api" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465618 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8114c66e-0643-42d4-9ee0-e9c9d1923852" containerName="istio-proxy" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465623 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1cf4d8-a4fe-4acc-a225-aaa71aece6d9" containerName="maas-api" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465628 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd52fc4-6175-4651-866f-c2afaec5c206" containerName="payload-pre-processing" Oct 31 03:57:55.465733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.465632 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fee47dc-7209-46db-8f3c-a27b016a114f" containerName="maas-api" Oct 31 03:57:55.470412 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.470396 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.527406 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.527367 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz"] Oct 31 03:57:55.573637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.573425 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:57:55.573637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.573457 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:57:55.639358 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.639322 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.639541 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.639506 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vkh\" (UniqueName: \"kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.639593 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.639569 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.740996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.740892 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vkh\" (UniqueName: \"kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.740996 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.740964 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.741342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.741032 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.741693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.741671 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.743556 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.743533 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:55.802966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:55.802931 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vkh\" (UniqueName: \"kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh\") pod \"maas-api-revoke-keys-e2e-api-b-b16de9-phbgz\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:56.084014 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:56.083985 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:56.465547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:56.465516 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz"] Oct 31 03:57:56.467150 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:57:56.467125 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e23fef4_81c9_4aea_80ef_372890aa8f38.slice/crio-41e0a9c06c61574856dc871c5c8a0316ea5d69c84c4420ceb5f5c6032e000ebf WatchSource:0}: Error finding container 41e0a9c06c61574856dc871c5c8a0316ea5d69c84c4420ceb5f5c6032e000ebf: Status 404 returned error can't find the container with id 41e0a9c06c61574856dc871c5c8a0316ea5d69c84c4420ceb5f5c6032e000ebf Oct 31 03:57:56.779594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:56.779562 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" event={"ID":"7e23fef4-81c9-4aea-80ef-372890aa8f38","Type":"ContainerStarted","Data":"41e0a9c06c61574856dc871c5c8a0316ea5d69c84c4420ceb5f5c6032e000ebf"} Oct 31 03:57:57.666511 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:57.666405 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:57.795553 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:57.795518 4488 generic.go:356] "Generic (PLEG): container finished" podID="7e23fef4-81c9-4aea-80ef-372890aa8f38" containerID="4a70bc8d5726cce3db1a7cf3c1f715b84e399befde5164e8297560434e2cc531" exitCode=0 Oct 31 03:57:57.795761 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:57.795610 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" event={"ID":"7e23fef4-81c9-4aea-80ef-372890aa8f38","Type":"ContainerDied","Data":"4a70bc8d5726cce3db1a7cf3c1f715b84e399befde5164e8297560434e2cc531"} Oct 31 03:57:58.960211 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.960184 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:58.982076 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.982041 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh\" (UniqueName: \"kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh\") pod \"7e23fef4-81c9-4aea-80ef-372890aa8f38\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " Oct 31 03:57:58.982311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.982254 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca\") pod \"7e23fef4-81c9-4aea-80ef-372890aa8f38\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " Oct 31 03:57:58.982360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.982340 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca\") pod \"7e23fef4-81c9-4aea-80ef-372890aa8f38\" (UID: \"7e23fef4-81c9-4aea-80ef-372890aa8f38\") " Oct 31 03:57:58.982675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.982646 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca" pod "7e23fef4-81c9-4aea-80ef-372890aa8f38" (UID: "7e23fef4-81c9-4aea-80ef-372890aa8f38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:58.982881 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.982863 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:58.985170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.985125 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca" pod "7e23fef4-81c9-4aea-80ef-372890aa8f38" (UID: "7e23fef4-81c9-4aea-80ef-372890aa8f38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:58.985291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:58.985206 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh" pod "7e23fef4-81c9-4aea-80ef-372890aa8f38" (UID: "7e23fef4-81c9-4aea-80ef-372890aa8f38"). InnerVolumeSpecName "kube-api-access-n6vkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:59.102112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:59.093283 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e23fef4-81c9-4aea-80ef-372890aa8f38-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:59.102112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:59.093325 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vkh\" (UniqueName: \"kubernetes.io/projected/7e23fef4-81c9-4aea-80ef-372890aa8f38-kube-api-access-n6vkh\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:57:59.354033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:59.353940 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:57:59.809079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:59.808989 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" event={"ID":"7e23fef4-81c9-4aea-80ef-372890aa8f38","Type":"ContainerDied","Data":"41e0a9c06c61574856dc871c5c8a0316ea5d69c84c4420ceb5f5c6032e000ebf"} Oct 31 03:57:59.809079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:59.809031 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz" Oct 31 03:57:59.809079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:57:59.809043 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e0a9c06c61574856dc871c5c8a0316ea5d69c84c4420ceb5f5c6032e000ebf" Oct 31 03:58:01.551383 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:01.551346 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:01.708439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:01.708408 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:02.928118 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:02.928051 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:06.480660 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.480623 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" containerID="cri-o://955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703" gracePeriod=45 Oct 31 03:58:06.553106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.552545 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": dial tcp 10.133.0.162:8000: connect: connection refused" Oct 31 03:58:06.592701 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:06.592665 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:58:06.747933 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.747851 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:58:06.838198 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838160 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir\") pod \"763a1185-9d80-411e-b963-6a16a9f19133\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " Oct 31 03:58:06.838390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838222 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw\" (UniqueName: \"kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw\") pod \"763a1185-9d80-411e-b963-6a16a9f19133\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " Oct 31 03:58:06.838439 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838392 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs\" (UniqueName: \"kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs\") pod \"763a1185-9d80-411e-b963-6a16a9f19133\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " Oct 31 03:58:06.838475 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838445 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm\") pod \"763a1185-9d80-411e-b963-6a16a9f19133\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " Oct 31 03:58:06.838517 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838485 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home\") pod \"763a1185-9d80-411e-b963-6a16a9f19133\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " Oct 31 03:58:06.838517 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838482 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir" pod "763a1185-9d80-411e-b963-6a16a9f19133" (UID: "763a1185-9d80-411e-b963-6a16a9f19133"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:06.838622 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838540 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache\") pod \"763a1185-9d80-411e-b963-6a16a9f19133\" (UID: \"763a1185-9d80-411e-b963-6a16a9f19133\") " Oct 31 03:58:06.838833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838770 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home" pod "763a1185-9d80-411e-b963-6a16a9f19133" (UID: "763a1185-9d80-411e-b963-6a16a9f19133"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:06.838833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.838824 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache" pod "763a1185-9d80-411e-b963-6a16a9f19133" (UID: "763a1185-9d80-411e-b963-6a16a9f19133"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:06.839273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.839035 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:06.839273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.839049 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:06.839273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.839074 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:06.840792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.840768 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm" pod "763a1185-9d80-411e-b963-6a16a9f19133" (UID: "763a1185-9d80-411e-b963-6a16a9f19133"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:06.840792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.840774 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs" pod "763a1185-9d80-411e-b963-6a16a9f19133" (UID: "763a1185-9d80-411e-b963-6a16a9f19133"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:06.840933 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.840777 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw" pod "763a1185-9d80-411e-b963-6a16a9f19133" (UID: "763a1185-9d80-411e-b963-6a16a9f19133"). InnerVolumeSpecName "kube-api-access-s59cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:06.854583 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.854555 4488 generic.go:356] "Generic (PLEG): container finished" podID="763a1185-9d80-411e-b963-6a16a9f19133" containerID="955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703" exitCode=0 Oct 31 03:58:06.854728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.854600 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" Oct 31 03:58:06.854728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.854651 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" event={"ID":"763a1185-9d80-411e-b963-6a16a9f19133","Type":"ContainerDied","Data":"955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703"} Oct 31 03:58:06.854728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.854705 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn" event={"ID":"763a1185-9d80-411e-b963-6a16a9f19133","Type":"ContainerDied","Data":"9e3d20cd1e06064c56e6bb6523105111a4b41afd5c9d05d78703073d05c84a08"} Oct 31 03:58:06.854728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.854727 4488 scope.go:122] "RemoveContainer" containerID="955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703" Oct 31 03:58:06.866676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.866657 4488 scope.go:122] "RemoveContainer" containerID="955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703" Oct 31 03:58:06.866966 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:06.866948 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703\": container with ID starting with 955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703 not found: ID does not exist" containerID="955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703" Oct 31 03:58:06.867039 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.866991 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703"} err="failed to get container status \"955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703\": rpc error: code = NotFound desc = could not find container \"955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703\": container with ID starting with 955bfe87db3b5a89477f824db8190d8c6778e442388b3da30eb6855b81127703 not found: ID does not exist" Oct 31 03:58:06.940702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.940670 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59cw\" (UniqueName: \"kubernetes.io/projected/763a1185-9d80-411e-b963-6a16a9f19133-kube-api-access-s59cw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:06.940702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.940700 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763a1185-9d80-411e-b963-6a16a9f19133-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:06.940702 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.940709 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763a1185-9d80-411e-b963-6a16a9f19133-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:06.980612 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.980578 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn"] Oct 31 03:58:06.984831 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:06.984803 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1180d818/e2e-lifecycle-model-1180d818-kserve-6888dcb569-fc9pn"] Oct 31 03:58:07.337248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:07.337194 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:07.496512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:07.495986 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:07.920350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:07.920313 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:08.316885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:08.316807 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763a1185-9d80-411e-b963-6a16a9f19133" path="/var/lib/kubelet/pods/763a1185-9d80-411e-b963-6a16a9f19133/volumes" Oct 31 03:58:11.495317 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:11.495285 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb"] Oct 31 03:58:11.495750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:11.495478 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" containerID="cri-o://3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216" gracePeriod=30 Oct 31 03:58:12.811403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:12.811372 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:16.554536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.554503 4488 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:58:16.554928 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.554564 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 31 03:58:16.782695 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.782666 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:58:16.880887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.880847 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.880887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.880895 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881006 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881045 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881080 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881274 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881303 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881330 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881381 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881341 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:16.881601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881397 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs\") pod \"0c47b296-063b-4ae9-beb1-27f6764afcdd\" (UID: \"0c47b296-063b-4ae9-beb1-27f6764afcdd\") " Oct 31 03:58:16.881601 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881393 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:16.881721 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881617 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:16.881820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881791 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:16.881877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.881833 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:16.882338 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.882311 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.882471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.882343 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c47b296-063b-4ae9-beb1-27f6764afcdd-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.882471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.882361 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.882471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.882375 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.882471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.882389 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.883560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.883537 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:16.883560 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.883555 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:16.883747 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.883591 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "kube-api-access-r8bsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:16.883830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.883815 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token" pod "0c47b296-063b-4ae9-beb1-27f6764afcdd" (UID: "0c47b296-063b-4ae9-beb1-27f6764afcdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:16.918266 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.918224 4488 generic.go:356] "Generic (PLEG): container finished" podID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerID="3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216" exitCode=0 Oct 31 03:58:16.918444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.918283 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" Oct 31 03:58:16.918444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.918285 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" event={"ID":"0c47b296-063b-4ae9-beb1-27f6764afcdd","Type":"ContainerDied","Data":"3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216"} Oct 31 03:58:16.918444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.918401 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb" event={"ID":"0c47b296-063b-4ae9-beb1-27f6764afcdd","Type":"ContainerDied","Data":"90a5d77488e976840374bbc7e216b69877b860a4467694a647ed609919b39db7"} Oct 31 03:58:16.918444 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.918428 4488 scope.go:122] "RemoveContainer" containerID="3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216" Oct 31 03:58:16.937883 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.937855 4488 scope.go:122] "RemoveContainer" containerID="3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216" Oct 31 03:58:16.938250 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:16.938228 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216\": container with ID starting with 3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216 not found: ID does not exist" containerID="3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216" Oct 31 03:58:16.938306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.938262 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216"} err="failed to get container status \"3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216\": rpc error: code = NotFound desc = could not find container \"3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216\": container with ID starting with 3ce168ca90d817813ec663a6f00d436ad375392f1e89c413574407fa4a5d2216 not found: ID does not exist" Oct 31 03:58:16.983800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.983754 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.983800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.983789 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.983800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.983799 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c47b296-063b-4ae9-beb1-27f6764afcdd-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:16.983800 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:16.983810 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bsc\" (UniqueName: \"kubernetes.io/projected/0c47b296-063b-4ae9-beb1-27f6764afcdd-kube-api-access-r8bsc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:17.069638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:17.069538 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb"] Oct 31 03:58:17.074295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:17.074263 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-slkpb"] Oct 31 03:58:17.604728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:17.604686 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:18.316683 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:18.316647 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" path="/var/lib/kubelet/pods/0c47b296-063b-4ae9-beb1-27f6764afcdd/volumes" Oct 31 03:58:19.696635 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:19.696602 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh"] Oct 31 03:58:19.697157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:19.696849 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" containerID="cri-o://d5dd09bb5b27859a91076a55a1d2c2a4bff341216f1b4eddae02127c6fe18664" gracePeriod=30 Oct 31 03:58:19.801737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:19.801703 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn"] Oct 31 03:58:19.801987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:19.801962 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" containerID="cri-o://22bdc596bd77d3960f2d4114b68c3d6c4bfb690a27b1d47722160c4a9c3549a2" gracePeriod=30 Oct 31 03:58:21.101430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:21.101390 4488 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:58:21.101829 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:21.101442 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:58:22.400081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:22.400037 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:24.975719 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.975683 4488 generic.go:356] "Generic (PLEG): container finished" podID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerID="22bdc596bd77d3960f2d4114b68c3d6c4bfb690a27b1d47722160c4a9c3549a2" exitCode=0 Oct 31 03:58:24.976227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.975831 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" event={"ID":"6eb50f54-f39c-4cac-bb0b-039339b0c778","Type":"ContainerDied","Data":"22bdc596bd77d3960f2d4114b68c3d6c4bfb690a27b1d47722160c4a9c3549a2"} Oct 31 03:58:24.976227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.975894 4488 scope.go:122] "RemoveContainer" containerID="006a8ea9486d30db3754ca42dff98287bd0c835d4ef209be5582687a405a7179" Oct 31 03:58:24.979716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.979649 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerID="d5dd09bb5b27859a91076a55a1d2c2a4bff341216f1b4eddae02127c6fe18664" exitCode=0 Oct 31 03:58:24.980003 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.979976 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" event={"ID":"6a18583d-f061-46bb-8af6-9b34518d4d95","Type":"ContainerDied","Data":"d5dd09bb5b27859a91076a55a1d2c2a4bff341216f1b4eddae02127c6fe18664"} Oct 31 03:58:24.980126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.980038 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" event={"ID":"6a18583d-f061-46bb-8af6-9b34518d4d95","Type":"ContainerDied","Data":"8199c6e8e1c3f330c0920c925ab4309c0e75a4970e1ac4d0649654afa93374ec"} Oct 31 03:58:24.980126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.980064 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8199c6e8e1c3f330c0920c925ab4309c0e75a4970e1ac4d0649654afa93374ec" Oct 31 03:58:24.992224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.992065 4488 scope.go:122] "RemoveContainer" containerID="efe25d3456b6e8dc2aa5c530ca14de386cf4b62538ad571a659f6e1166321c5e" Oct 31 03:58:24.994182 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:24.992617 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:58:25.090476 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090446 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090497 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090535 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090644 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090638 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090688 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090711 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.090937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090734 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090796 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.090937 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090897 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.091321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.090931 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.091321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.091124 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data\") pod \"6a18583d-f061-46bb-8af6-9b34518d4d95\" (UID: \"6a18583d-f061-46bb-8af6-9b34518d4d95\") " Oct 31 03:58:25.091321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.091184 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:25.091771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.091393 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.091771 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.091413 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.092586 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.092559 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.092713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.092590 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.092713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.092606 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.092713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.092621 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18583d-f061-46bb-8af6-9b34518d4d95-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.092713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.092635 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.093451 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.093421 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:25.093993 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.093963 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:25.094716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.094694 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "kube-api-access-kdj7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:25.094842 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.094821 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy" pod "6a18583d-f061-46bb-8af6-9b34518d4d95" (UID: "6a18583d-f061-46bb-8af6-9b34518d4d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.104173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.104152 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:58:25.193564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193530 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193607 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193658 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193758 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193786 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193838 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193917 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.193966 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193956 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.194224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.193981 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.194224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194050 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert\") pod \"6eb50f54-f39c-4cac-bb0b-039339b0c778\" (UID: \"6eb50f54-f39c-4cac-bb0b-039339b0c778\") " Oct 31 03:58:25.194224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194208 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.194371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194230 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.194371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194281 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.194832 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194798 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194837 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194859 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194874 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194889 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18583d-f061-46bb-8af6-9b34518d4d95-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194907 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194920 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194934 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj7b\" (UniqueName: \"kubernetes.io/projected/6a18583d-f061-46bb-8af6-9b34518d4d95-kube-api-access-kdj7b\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.194957 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.194949 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.197119 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.197066 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:25.197396 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.197371 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:25.198006 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.197982 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "kube-api-access-4579f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:25.198077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.198063 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy" pod "6eb50f54-f39c-4cac-bb0b-039339b0c778" (UID: "6eb50f54-f39c-4cac-bb0b-039339b0c778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:25.296890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.296800 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.296890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.296826 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.296890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.296835 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4579f\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-kube-api-access-4579f\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.296890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.296846 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb50f54-f39c-4cac-bb0b-039339b0c778-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.296890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.296858 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb50f54-f39c-4cac-bb0b-039339b0c778-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:25.987332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.987249 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" Oct 31 03:58:25.987332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.987295 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn" event={"ID":"6eb50f54-f39c-4cac-bb0b-039339b0c778","Type":"ContainerDied","Data":"eb0aebe6a00578d868e4a4afec32dbd610748cc465c55517343f522b69a90fee"} Oct 31 03:58:25.987850 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.987357 4488 scope.go:122] "RemoveContainer" containerID="22bdc596bd77d3960f2d4114b68c3d6c4bfb690a27b1d47722160c4a9c3549a2" Oct 31 03:58:25.988907 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:25.988888 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh" Oct 31 03:58:26.203942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:26.203900 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn"] Oct 31 03:58:26.253782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:26.253698 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-drbrn"] Oct 31 03:58:26.317718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:26.317688 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" path="/var/lib/kubelet/pods/6eb50f54-f39c-4cac-bb0b-039339b0c778/volumes" Oct 31 03:58:26.386269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:26.386232 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh"] Oct 31 03:58:26.390484 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:26.390453 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-fpmkh"] Oct 31 03:58:26.589027 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:26.588997 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:27.910877 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:27.910845 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:28.192833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:28.192742 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:28.317006 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:28.316963 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" path="/var/lib/kubelet/pods/6a18583d-f061-46bb-8af6-9b34518d4d95/volumes" Oct 31 03:58:28.501867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:28.501781 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:28.778509 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:28.778422 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:29.151637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:29.151610 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:29.412194 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:29.412076 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:30.358050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:30.358020 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:32.886987 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:32.886956 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.057127 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:35.057065 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.273035 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:35.273005 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.321944 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:35.321844 4488 scope.go:122] "RemoveContainer" containerID="d5dd09bb5b27859a91076a55a1d2c2a4bff341216f1b4eddae02127c6fe18664" Oct 31 03:58:35.427809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:35.427776 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.591562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:35.591521 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.911448 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:35.911304 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:36.066998 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.066965 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:36.342836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.342804 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:36.500310 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.500277 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:36.591029 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:36.590995 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:58:36.670111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.670012 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm"] Oct 31 03:58:36.670287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.670241 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" containerID="cri-o://78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0" gracePeriod=30 Oct 31 03:58:36.680637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.680607 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h"] Oct 31 03:58:36.680850 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.680830 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" containerID="cri-o://b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c" gracePeriod=30 Oct 31 03:58:36.682248 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.682224 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:36.705994 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.705960 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q"] Oct 31 03:58:36.706246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.706223 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" containerID="cri-o://5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206" gracePeriod=30 Oct 31 03:58:36.851818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.851767 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf"] Oct 31 03:58:36.852269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852255 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852270 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852280 4488 cpu_manager.go:414] "containerMap: removing container" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852286 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852296 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852302 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852316 4488 cpu_manager.go:414] "containerMap: removing container" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" Oct 31 03:58:36.852322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852321 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852332 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852336 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852344 4488 cpu_manager.go:414] "containerMap: removing container" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852349 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852362 4488 cpu_manager.go:414] "containerMap: removing container" podUID="7e23fef4-81c9-4aea-80ef-372890aa8f38" containerName="revoke-keys" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852366 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e23fef4-81c9-4aea-80ef-372890aa8f38" containerName="revoke-keys" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852373 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852377 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7b08bd-1c06-4308-84ed-f71ef4043463" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852383 4488 cpu_manager.go:414] "containerMap: removing container" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852388 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852468 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852478 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="763a1185-9d80-411e-b963-6a16a9f19133" containerName="main" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852483 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852488 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c5632c-d89d-4db0-a838-b5b40db09ee8" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852495 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="228357ef-0268-409f-bd42-73f83b0567de" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852501 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a18583d-f061-46bb-8af6-9b34518d4d95" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852505 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c47b296-063b-4ae9-beb1-27f6764afcdd" containerName="istio-proxy" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852511 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e23fef4-81c9-4aea-80ef-372890aa8f38" containerName="revoke-keys" Oct 31 03:58:36.852547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.852519 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" Oct 31 03:58:36.857382 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.857361 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.868594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.868566 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.868726 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.868628 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.868811 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.868790 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqjk\" (UniqueName: \"kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.912946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.912917 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf"] Oct 31 03:58:36.913193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.913169 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:36.914025 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.914004 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:58:36.970374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.970273 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqjk\" (UniqueName: \"kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.970374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.970336 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.970374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.970371 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.971011 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.970990 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:36.972882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:36.972853 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:37.031539 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:37.031512 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqjk\" (UniqueName: \"kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk\") pod \"maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:37.171677 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:37.171645 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:37.552099 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:37.552061 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf"] Oct 31 03:58:37.553915 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:58:37.553887 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0dca2c_863f_4003_9f03_6bdad394b110.slice/crio-5fae9f3ba7033e8ed4aa083a5adcf0ba898551267a5cb49cfcbddb746bb6c300 WatchSource:0}: Error finding container 5fae9f3ba7033e8ed4aa083a5adcf0ba898551267a5cb49cfcbddb746bb6c300: Status 404 returned error can't find the container with id 5fae9f3ba7033e8ed4aa083a5adcf0ba898551267a5cb49cfcbddb746bb6c300 Oct 31 03:58:38.075856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:38.075739 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" event={"ID":"1d0dca2c-863f-4003-9f03-6bdad394b110","Type":"ContainerStarted","Data":"0b66cc12496bc01aa06bd1256be29eb3c992bfb3652d3723b3c266406c762893"} Oct 31 03:58:38.075856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:38.075812 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" event={"ID":"1d0dca2c-863f-4003-9f03-6bdad394b110","Type":"ContainerStarted","Data":"5fae9f3ba7033e8ed4aa083a5adcf0ba898551267a5cb49cfcbddb746bb6c300"} Oct 31 03:58:38.192395 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:38.192339 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" podStartSLOduration=2.192324253 podStartE2EDuration="2.192324253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:38.191445956 +0000 UTC m=+2345.988164226" watchObservedRunningTime="2026-10-31 03:58:38.192324253 +0000 UTC m=+2345.989042512" Oct 31 03:58:39.082701 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:39.082668 4488 generic.go:356] "Generic (PLEG): container finished" podID="1d0dca2c-863f-4003-9f03-6bdad394b110" containerID="0b66cc12496bc01aa06bd1256be29eb3c992bfb3652d3723b3c266406c762893" exitCode=0 Oct 31 03:58:39.082898 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:39.082748 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" event={"ID":"1d0dca2c-863f-4003-9f03-6bdad394b110","Type":"ContainerDied","Data":"0b66cc12496bc01aa06bd1256be29eb3c992bfb3652d3723b3c266406c762893"} Oct 31 03:58:40.156446 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.156399 4488 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:58:40.156821 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.156510 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:58:40.244106 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.244065 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:40.309989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.309953 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk\" (UniqueName: \"kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk\") pod \"1d0dca2c-863f-4003-9f03-6bdad394b110\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " Oct 31 03:58:40.309989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.310002 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca\") pod \"1d0dca2c-863f-4003-9f03-6bdad394b110\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " Oct 31 03:58:40.310369 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.310347 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca" pod "1d0dca2c-863f-4003-9f03-6bdad394b110" (UID: "1d0dca2c-863f-4003-9f03-6bdad394b110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:40.310437 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.310377 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca\") pod \"1d0dca2c-863f-4003-9f03-6bdad394b110\" (UID: \"1d0dca2c-863f-4003-9f03-6bdad394b110\") " Oct 31 03:58:40.310751 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.310731 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:40.312458 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.312426 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk" pod "1d0dca2c-863f-4003-9f03-6bdad394b110" (UID: "1d0dca2c-863f-4003-9f03-6bdad394b110"). InnerVolumeSpecName "kube-api-access-qzqjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:40.312564 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.312470 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca" pod "1d0dca2c-863f-4003-9f03-6bdad394b110" (UID: "1d0dca2c-863f-4003-9f03-6bdad394b110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:40.412542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.412458 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqjk\" (UniqueName: \"kubernetes.io/projected/1d0dca2c-863f-4003-9f03-6bdad394b110-kube-api-access-qzqjk\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:40.412542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.412488 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0dca2c-863f-4003-9f03-6bdad394b110-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:40.548047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:40.548015 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:41.017415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.017384 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:41.094809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.094778 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" event={"ID":"1d0dca2c-863f-4003-9f03-6bdad394b110","Type":"ContainerDied","Data":"5fae9f3ba7033e8ed4aa083a5adcf0ba898551267a5cb49cfcbddb746bb6c300"} Oct 31 03:58:41.094809 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.094812 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf" Oct 31 03:58:41.095032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.094834 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fae9f3ba7033e8ed4aa083a5adcf0ba898551267a5cb49cfcbddb746bb6c300" Oct 31 03:58:41.315939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.315842 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg"] Oct 31 03:58:41.326268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.326238 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr"] Oct 31 03:58:41.410443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.410405 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:41.976755 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:41.976729 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:58:42.027816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.027788 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:58:42.031665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031642 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.031806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031717 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.031806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031739 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.031806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031772 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.031806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031792 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.032057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031810 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.032057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031889 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.032057 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.031930 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.032268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032057 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.032268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032117 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.032268 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032148 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo\") pod \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\" (UID: \"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0\") " Oct 31 03:58:42.032483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032141 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.032483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032387 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:58:42.032483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032299 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.032483 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032350 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:42.032949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032928 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.033018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032955 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.033018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032973 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.033018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.032987 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.033018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.033002 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.034172 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.034152 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.034353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.034334 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:42.034820 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.034800 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.034967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.034952 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb" pod "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" (UID: "5826cb5a-f4d1-4fcb-84aa-eda4866fcff0"). InnerVolumeSpecName "kube-api-access-p2fqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.106007 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.105972 4488 generic.go:356] "Generic (PLEG): container finished" podID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerID="b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c" exitCode=0 Oct 31 03:58:42.106231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.106021 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" Oct 31 03:58:42.106231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.106063 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" event={"ID":"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc","Type":"ContainerDied","Data":"b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c"} Oct 31 03:58:42.106231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.106168 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h" event={"ID":"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc","Type":"ContainerDied","Data":"8c4efd9dfd8f66506c8ec757f7a12c3d939dffe805875ebbbf6874ad73a180a2"} Oct 31 03:58:42.106231 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.106199 4488 scope.go:122] "RemoveContainer" containerID="b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c" Oct 31 03:58:42.108699 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.108676 4488 generic.go:356] "Generic (PLEG): container finished" podID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerID="78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0" exitCode=0 Oct 31 03:58:42.108849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.108751 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" Oct 31 03:58:42.108849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.108767 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" event={"ID":"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0","Type":"ContainerDied","Data":"78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0"} Oct 31 03:58:42.108849 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.108821 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm" event={"ID":"5826cb5a-f4d1-4fcb-84aa-eda4866fcff0","Type":"ContainerDied","Data":"e9159926c60f979854a4db1e75073be3a45757718035a048f6ad0ed696b7b352"} Oct 31 03:58:42.111269 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.111245 4488 generic.go:356] "Generic (PLEG): container finished" podID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerID="5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206" exitCode=0 Oct 31 03:58:42.111471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.111286 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" event={"ID":"7d41da34-9b31-4d0d-a545-770e5a1d6e82","Type":"ContainerDied","Data":"5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206"} Oct 31 03:58:42.111471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.111312 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" Oct 31 03:58:42.111471 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.111326 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q" event={"ID":"7d41da34-9b31-4d0d-a545-770e5a1d6e82","Type":"ContainerDied","Data":"a858b86fbd8a3bfc241c22031df576cd6c03b543cf171e8b4c4e9d39ce33d1a4"} Oct 31 03:58:42.117627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.117459 4488 scope.go:122] "RemoveContainer" containerID="b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c" Oct 31 03:58:42.117885 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:42.117860 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c\": container with ID starting with b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c not found: ID does not exist" containerID="b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c" Oct 31 03:58:42.117985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.117898 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c"} err="failed to get container status \"b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c\": rpc error: code = NotFound desc = could not find container \"b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c\": container with ID starting with b25878da4aa19cea39ce2453ba63b9d813b5eeaa90d5566eed1840698bd9229c not found: ID does not exist" Oct 31 03:58:42.117985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.117920 4488 scope.go:122] "RemoveContainer" containerID="78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0" Oct 31 03:58:42.127510 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.127491 4488 scope.go:122] "RemoveContainer" containerID="78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0" Oct 31 03:58:42.127836 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:42.127817 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0\": container with ID starting with 78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0 not found: ID does not exist" containerID="78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0" Oct 31 03:58:42.127927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.127849 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0"} err="failed to get container status \"78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0\": rpc error: code = NotFound desc = could not find container \"78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0\": container with ID starting with 78dc8c24db1f43e07f63942aa2f2d430bb9f31fd4e1876015925cb4d9942c4a0 not found: ID does not exist" Oct 31 03:58:42.127927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.127875 4488 scope.go:122] "RemoveContainer" containerID="5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206" Oct 31 03:58:42.138688 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.138662 4488 scope.go:122] "RemoveContainer" containerID="14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b" Oct 31 03:58:42.139277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139145 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.139277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139231 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.139277 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139266 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.139491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139295 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.139491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139337 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.139491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139379 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.139491 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139422 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.139696 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.139587 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:42.140176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140157 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.140286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140191 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.140286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140223 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.140286 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140257 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.140452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140315 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.140452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140342 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.140452 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140423 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.140623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140460 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.140623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140502 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket\") pod \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\" (UID: \"8b9b9f45-bc13-4fcf-9c0d-3517560e10dc\") " Oct 31 03:58:42.140623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140535 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.140623 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.140571 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token\") pod \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\" (UID: \"7d41da34-9b31-4d0d-a545-770e5a1d6e82\") " Oct 31 03:58:42.141300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141251 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.141300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141276 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.141300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141292 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.141300 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141305 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.141609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141319 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fqb\" (UniqueName: \"kubernetes.io/projected/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0-kube-api-access-p2fqb\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.141609 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141386 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.141713 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141664 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.141897 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.141874 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.142239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.142213 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.142562 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.142472 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.143252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.142733 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.143252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.142962 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.143252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.143001 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.143252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.143200 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:42.143835 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.143608 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.144262 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.144227 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:42.144682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.144662 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "kube-api-access-lwst7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.144848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.144825 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.144948 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.144868 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.145567 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.145547 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j" pod "7d41da34-9b31-4d0d-a545-770e5a1d6e82" (UID: "7d41da34-9b31-4d0d-a545-770e5a1d6e82"). InnerVolumeSpecName "kube-api-access-fwj4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:42.146077 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.146057 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:58:42.146637 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.146621 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy" pod "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" (UID: "8b9b9f45-bc13-4fcf-9c0d-3517560e10dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:42.153710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.153689 4488 scope.go:122] "RemoveContainer" containerID="5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206" Oct 31 03:58:42.154078 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:42.154056 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206\": container with ID starting with 5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206 not found: ID does not exist" containerID="5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206" Oct 31 03:58:42.154175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.154124 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206"} err="failed to get container status \"5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206\": rpc error: code = NotFound desc = could not find container \"5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206\": container with ID starting with 5c52048b56469b2744a316b92bc5a8a2cb6ad7dde045579ce6f9ecc2422d4206 not found: ID does not exist" Oct 31 03:58:42.154175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.154152 4488 scope.go:122] "RemoveContainer" containerID="14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b" Oct 31 03:58:42.154480 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:42.154455 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b\": container with ID starting with 14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b not found: ID does not exist" containerID="14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b" Oct 31 03:58:42.154595 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.154486 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b"} err="failed to get container status \"14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b\": rpc error: code = NotFound desc = could not find container \"14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b\": container with ID starting with 14965aa9747aca31569c43f6f5f542cc8195f747fb4449c647225a90c5d7f61b not found: ID does not exist" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242433 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242464 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242474 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj4j\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-kube-api-access-fwj4j\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242483 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242492 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242500 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242509 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242516 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242524 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.242522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242532 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwst7\" (UniqueName: \"kubernetes.io/projected/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-kube-api-access-lwst7\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242541 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242550 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242559 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242566 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242574 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242583 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.243213 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.242592 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d41da34-9b31-4d0d-a545-770e5a1d6e82-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:42.506455 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.506371 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm"] Oct 31 03:58:42.513176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.513132 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-67hlm"] Oct 31 03:58:42.674950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.674917 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q"] Oct 31 03:58:42.681155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.681113 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lzk6q"] Oct 31 03:58:42.719571 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.719528 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:42.790836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.790749 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h"] Oct 31 03:58:42.794776 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.794751 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-65h6h"] Oct 31 03:58:42.885808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:42.885768 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:43.068989 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:43.068877 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:43.572233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:43.572200 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.052365 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt"] Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.052971 4488 cpu_manager.go:414] "containerMap: removing container" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.052986 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053007 4488 cpu_manager.go:414] "containerMap: removing container" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053014 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053029 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053036 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb50f54-f39c-4cac-bb0b-039339b0c778" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053052 4488 cpu_manager.go:414] "containerMap: removing container" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053059 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053080 4488 cpu_manager.go:414] "containerMap: removing container" podUID="1d0dca2c-863f-4003-9f03-6bdad394b110" containerName="revoke-keys" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053106 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0dca2c-863f-4003-9f03-6bdad394b110" containerName="revoke-keys" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053130 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053136 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053241 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0dca2c-863f-4003-9f03-6bdad394b110" containerName="revoke-keys" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053253 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053268 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053276 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" containerName="istio-proxy" Oct 31 03:58:44.055068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.053287 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" containerName="istio-proxy" Oct 31 03:58:44.059115 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.059080 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.114949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.114910 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt"] Oct 31 03:58:44.161332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.161303 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:44.161513 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.161378 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:58:44.167942 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.167921 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.168033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.167970 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.168155 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.168138 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlq8\" (UniqueName: \"kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.269005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.268968 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlq8\" (UniqueName: \"kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.269220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.269181 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.269220 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.269210 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.269822 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.269790 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.271716 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.271693 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.316366 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.316287 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5826cb5a-f4d1-4fcb-84aa-eda4866fcff0" path="/var/lib/kubelet/pods/5826cb5a-f4d1-4fcb-84aa-eda4866fcff0/volumes" Oct 31 03:58:44.316774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.316761 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d41da34-9b31-4d0d-a545-770e5a1d6e82" path="/var/lib/kubelet/pods/7d41da34-9b31-4d0d-a545-770e5a1d6e82/volumes" Oct 31 03:58:44.317210 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.317197 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9b9f45-bc13-4fcf-9c0d-3517560e10dc" path="/var/lib/kubelet/pods/8b9b9f45-bc13-4fcf-9c0d-3517560e10dc/volumes" Oct 31 03:58:44.330447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.330411 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlq8\" (UniqueName: \"kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8\") pod \"maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.380520 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.380484 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:44.761133 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.761107 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt"] Oct 31 03:58:44.762561 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:58:44.762525 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9026167_3bd6_4a76_83ce_ae2347fb41ec.slice/crio-b23fdef24f10d30153d201a208058b2b687321897e98a6d3f0bf3139a3f9d7bd WatchSource:0}: Error finding container b23fdef24f10d30153d201a208058b2b687321897e98a6d3f0bf3139a3f9d7bd: Status 404 returned error can't find the container with id b23fdef24f10d30153d201a208058b2b687321897e98a6d3f0bf3139a3f9d7bd Oct 31 03:58:44.866946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:44.866910 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:58:45.163516 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:45.163485 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" event={"ID":"f9026167-3bd6-4a76-83ce-ae2347fb41ec","Type":"ContainerStarted","Data":"c8f20ab07ede83dc186dfacbfa2126e05e1c0a1fd9e300b60a8acf1e9ecd57c6"} Oct 31 03:58:45.163710 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:45.163536 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" event={"ID":"f9026167-3bd6-4a76-83ce-ae2347fb41ec","Type":"ContainerStarted","Data":"b23fdef24f10d30153d201a208058b2b687321897e98a6d3f0bf3139a3f9d7bd"} Oct 31 03:58:45.280887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:45.280766 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" podStartSLOduration=1.280748515 podStartE2EDuration="1.280748515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:45.27913699 +0000 UTC m=+2353.075855245" watchObservedRunningTime="2026-10-31 03:58:45.280748515 +0000 UTC m=+2353.077466821" Oct 31 03:58:45.645154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:45.645075 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:46.170499 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:46.170467 4488 generic.go:356] "Generic (PLEG): container finished" podID="f9026167-3bd6-4a76-83ce-ae2347fb41ec" containerID="c8f20ab07ede83dc186dfacbfa2126e05e1c0a1fd9e300b60a8acf1e9ecd57c6" exitCode=0 Oct 31 03:58:46.170930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:46.170556 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" event={"ID":"f9026167-3bd6-4a76-83ce-ae2347fb41ec","Type":"ContainerDied","Data":"c8f20ab07ede83dc186dfacbfa2126e05e1c0a1fd9e300b60a8acf1e9ecd57c6"} Oct 31 03:58:47.321768 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.321742 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:47.408687 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.408648 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca\") pod \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " Oct 31 03:58:47.408899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.408761 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca\") pod \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " Oct 31 03:58:47.408899 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.408856 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8\" (UniqueName: \"kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8\") pod \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\" (UID: \"f9026167-3bd6-4a76-83ce-ae2347fb41ec\") " Oct 31 03:58:47.409165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.409120 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca" pod "f9026167-3bd6-4a76-83ce-ae2347fb41ec" (UID: "f9026167-3bd6-4a76-83ce-ae2347fb41ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:47.410960 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.410934 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca" pod "f9026167-3bd6-4a76-83ce-ae2347fb41ec" (UID: "f9026167-3bd6-4a76-83ce-ae2347fb41ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:47.411154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.411071 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8" pod "f9026167-3bd6-4a76-83ce-ae2347fb41ec" (UID: "f9026167-3bd6-4a76-83ce-ae2347fb41ec"). InnerVolumeSpecName "kube-api-access-zwlq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:47.509927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.509847 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:47.509927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.509873 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9026167-3bd6-4a76-83ce-ae2347fb41ec-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:47.509927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:47.509885 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwlq8\" (UniqueName: \"kubernetes.io/projected/f9026167-3bd6-4a76-83ce-ae2347fb41ec-kube-api-access-zwlq8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.183786 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.183754 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" Oct 31 03:58:48.184018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.183744 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt" event={"ID":"f9026167-3bd6-4a76-83ce-ae2347fb41ec","Type":"ContainerDied","Data":"b23fdef24f10d30153d201a208058b2b687321897e98a6d3f0bf3139a3f9d7bd"} Oct 31 03:58:48.184018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.183893 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23fdef24f10d30153d201a208058b2b687321897e98a6d3f0bf3139a3f9d7bd" Oct 31 03:58:48.540857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.540756 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf"] Oct 31 03:58:48.648000 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.647907 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:48.810247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.805594 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:48.968010 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:48.967971 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:50.799179 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:50.799146 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:55.603334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:55.603278 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:55.934967 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:55.934880 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:56.092512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.092461 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:56.337457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.337423 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" containerID="cri-o://0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea" gracePeriod=45 Oct 31 03:58:56.353617 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.353583 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" containerID="cri-o://fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa" gracePeriod=45 Oct 31 03:58:56.630470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.630443 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:58:56.656239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.656213 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:58:56.723050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723023 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh\" (UniqueName: \"kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh\") pod \"f468edbd-35ea-4024-8dcd-36386b294e5a\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " Oct 31 03:58:56.723236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723078 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs\") pod \"f468edbd-35ea-4024-8dcd-36386b294e5a\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " Oct 31 03:58:56.723236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723160 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache\") pod \"f468edbd-35ea-4024-8dcd-36386b294e5a\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " Oct 31 03:58:56.723236 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723213 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir\") pod \"f468edbd-35ea-4024-8dcd-36386b294e5a\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " Oct 31 03:58:56.723418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723302 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm\") pod \"f468edbd-35ea-4024-8dcd-36386b294e5a\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " Oct 31 03:58:56.723418 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723335 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home\") pod \"f468edbd-35ea-4024-8dcd-36386b294e5a\" (UID: \"f468edbd-35ea-4024-8dcd-36386b294e5a\") " Oct 31 03:58:56.723964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723549 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir" pod "f468edbd-35ea-4024-8dcd-36386b294e5a" (UID: "f468edbd-35ea-4024-8dcd-36386b294e5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.723964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723796 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home" pod "f468edbd-35ea-4024-8dcd-36386b294e5a" (UID: "f468edbd-35ea-4024-8dcd-36386b294e5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.723964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723806 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache" pod "f468edbd-35ea-4024-8dcd-36386b294e5a" (UID: "f468edbd-35ea-4024-8dcd-36386b294e5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.723964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723937 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.723964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723970 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.723964 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.723980 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.726131 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.726080 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs" pod "f468edbd-35ea-4024-8dcd-36386b294e5a" (UID: "f468edbd-35ea-4024-8dcd-36386b294e5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:56.726360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.726337 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh" pod "f468edbd-35ea-4024-8dcd-36386b294e5a" (UID: "f468edbd-35ea-4024-8dcd-36386b294e5a"). InnerVolumeSpecName "kube-api-access-qt5dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:56.726470 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.726353 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm" pod "f468edbd-35ea-4024-8dcd-36386b294e5a" (UID: "f468edbd-35ea-4024-8dcd-36386b294e5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.824986 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.824936 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs\") pod \"a2958864-adde-4e3c-9ad8-48a47fb83f34\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " Oct 31 03:58:56.825216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825018 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir\") pod \"a2958864-adde-4e3c-9ad8-48a47fb83f34\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " Oct 31 03:58:56.825216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825047 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7\" (UniqueName: \"kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7\") pod \"a2958864-adde-4e3c-9ad8-48a47fb83f34\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " Oct 31 03:58:56.825216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825081 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache\") pod \"a2958864-adde-4e3c-9ad8-48a47fb83f34\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " Oct 31 03:58:56.825216 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825180 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home\") pod \"a2958864-adde-4e3c-9ad8-48a47fb83f34\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " Oct 31 03:58:56.825500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825285 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm\") pod \"a2958864-adde-4e3c-9ad8-48a47fb83f34\" (UID: \"a2958864-adde-4e3c-9ad8-48a47fb83f34\") " Oct 31 03:58:56.825500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825345 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir" pod "a2958864-adde-4e3c-9ad8-48a47fb83f34" (UID: "a2958864-adde-4e3c-9ad8-48a47fb83f34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.825500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825406 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache" pod "a2958864-adde-4e3c-9ad8-48a47fb83f34" (UID: "a2958864-adde-4e3c-9ad8-48a47fb83f34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.825500 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825492 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home" pod "a2958864-adde-4e3c-9ad8-48a47fb83f34" (UID: "a2958864-adde-4e3c-9ad8-48a47fb83f34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.825781 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825764 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468edbd-35ea-4024-8dcd-36386b294e5a-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.825851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825786 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.825851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825799 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.825851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825812 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.825851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825827 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt5dh\" (UniqueName: \"kubernetes.io/projected/f468edbd-35ea-4024-8dcd-36386b294e5a-kube-api-access-qt5dh\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.825851 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.825840 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468edbd-35ea-4024-8dcd-36386b294e5a-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.827415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.827393 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm" pod "a2958864-adde-4e3c-9ad8-48a47fb83f34" (UID: "a2958864-adde-4e3c-9ad8-48a47fb83f34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:58:56.827473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.827436 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs" pod "a2958864-adde-4e3c-9ad8-48a47fb83f34" (UID: "a2958864-adde-4e3c-9ad8-48a47fb83f34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:56.827473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.827446 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7" pod "a2958864-adde-4e3c-9ad8-48a47fb83f34" (UID: "a2958864-adde-4e3c-9ad8-48a47fb83f34"). InnerVolumeSpecName "kube-api-access-ztdk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:56.926841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.926758 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2958864-adde-4e3c-9ad8-48a47fb83f34-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.926841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.926797 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztdk7\" (UniqueName: \"kubernetes.io/projected/a2958864-adde-4e3c-9ad8-48a47fb83f34-kube-api-access-ztdk7\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:56.926841 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:56.926814 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2958864-adde-4e3c-9ad8-48a47fb83f34-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:58:57.245394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.245294 4488 generic.go:356] "Generic (PLEG): container finished" podID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerID="0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea" exitCode=0 Oct 31 03:58:57.245394 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.245338 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" Oct 31 03:58:57.245641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.245381 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" event={"ID":"f468edbd-35ea-4024-8dcd-36386b294e5a","Type":"ContainerDied","Data":"0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea"} Oct 31 03:58:57.245641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.245433 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" event={"ID":"f468edbd-35ea-4024-8dcd-36386b294e5a","Type":"ContainerDied","Data":"d6702612a16eebd7f020284cee3530675a3805c2a08709d0eeeb86d034858792"} Oct 31 03:58:57.245641 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.245456 4488 scope.go:122] "RemoveContainer" containerID="0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea" Oct 31 03:58:57.247195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.247175 4488 generic.go:356] "Generic (PLEG): container finished" podID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerID="fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa" exitCode=0 Oct 31 03:58:57.247316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.247264 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" event={"ID":"a2958864-adde-4e3c-9ad8-48a47fb83f34","Type":"ContainerDied","Data":"fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa"} Oct 31 03:58:57.247316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.247292 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" event={"ID":"a2958864-adde-4e3c-9ad8-48a47fb83f34","Type":"ContainerDied","Data":"5e3fe5c9d32e88193acd9b26150efc53aa69ca5a29aa1e8b3f8472e914540ed7"} Oct 31 03:58:57.247316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.247299 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr" Oct 31 03:58:57.255136 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.255118 4488 scope.go:122] "RemoveContainer" containerID="0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea" Oct 31 03:58:57.255456 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:57.255435 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea\": container with ID starting with 0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea not found: ID does not exist" containerID="0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea" Oct 31 03:58:57.255547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.255464 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea"} err="failed to get container status \"0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea\": rpc error: code = NotFound desc = could not find container \"0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea\": container with ID starting with 0c57f2744f7a6c2e2a0d056cfb6df4cfd8ca4d1584d6daf5844e2b6640157eea not found: ID does not exist" Oct 31 03:58:57.255547 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.255483 4488 scope.go:122] "RemoveContainer" containerID="fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa" Oct 31 03:58:57.264751 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.264729 4488 scope.go:122] "RemoveContainer" containerID="fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa" Oct 31 03:58:57.265075 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:58:57.265050 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa\": container with ID starting with fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa not found: ID does not exist" containerID="fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa" Oct 31 03:58:57.265208 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.265080 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa"} err="failed to get container status \"fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa\": rpc error: code = NotFound desc = could not find container \"fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa\": container with ID starting with fab6254100138ed5194556013fbf3cb484632317273ce5012a617f6573375baa not found: ID does not exist" Oct 31 03:58:57.409521 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.409492 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg"] Oct 31 03:58:57.413962 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.413929 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg"] Oct 31 03:58:57.565295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.559370 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr"] Oct 31 03:58:57.577334 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.577299 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/simulator-w6-b63425-kserve-5c474c9f5f-mvlqr"] Oct 31 03:58:57.579040 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:57.579007 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/premium-w6-b63425-kserve-56dc655b6-l7kkg" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:58:58.316815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:58.316776 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" path="/var/lib/kubelet/pods/a2958864-adde-4e3c-9ad8-48a47fb83f34/volumes" Oct 31 03:58:58.317255 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:58.317241 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" path="/var/lib/kubelet/pods/f468edbd-35ea-4024-8dcd-36386b294e5a/volumes" Oct 31 03:58:59.908815 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:58:59.908774 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:00.185225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:00.185114 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:00.340522 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:00.340486 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:01.082801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.082770 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc"] Oct 31 03:59:01.083249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083215 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" Oct 31 03:59:01.083249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083226 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" Oct 31 03:59:01.083249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083240 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" Oct 31 03:59:01.083249 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083245 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" Oct 31 03:59:01.083385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083254 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f9026167-3bd6-4a76-83ce-ae2347fb41ec" containerName="revoke-keys" Oct 31 03:59:01.083385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083259 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9026167-3bd6-4a76-83ce-ae2347fb41ec" containerName="revoke-keys" Oct 31 03:59:01.083385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083325 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2958864-adde-4e3c-9ad8-48a47fb83f34" containerName="main" Oct 31 03:59:01.083385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083334 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9026167-3bd6-4a76-83ce-ae2347fb41ec" containerName="revoke-keys" Oct 31 03:59:01.083385 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.083346 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f468edbd-35ea-4024-8dcd-36386b294e5a" containerName="main" Oct 31 03:59:01.086389 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.086374 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.144501 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.144467 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc"] Oct 31 03:59:01.227606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.227578 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:59:01.227606 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.227596 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:01.275050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.275017 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.275050 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.275054 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.275306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.275113 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvz6\" (UniqueName: \"kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.376638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.376548 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.376638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.376591 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.376638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.376629 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvz6\" (UniqueName: \"kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.377326 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.377298 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.379101 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.379062 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.437904 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.437871 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvz6\" (UniqueName: \"kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6\") pod \"maas-api-revoke-sub-8c7f59a5-cp9dc\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.461350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.461322 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:01.699542 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.699455 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:01.742757 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.742721 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:01.921171 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:01.921127 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:02.080271 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:02.080243 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc"] Oct 31 03:59:02.082401 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:02.082370 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca09c97_0fd2_490b_8a4b_ef50aa274650.slice/crio-5ddd54264198f2a0d0f034b3453416fc127f19888a8f85671c503cd0c78a8776 WatchSource:0}: Error finding container 5ddd54264198f2a0d0f034b3453416fc127f19888a8f85671c503cd0c78a8776: Status 404 returned error can't find the container with id 5ddd54264198f2a0d0f034b3453416fc127f19888a8f85671c503cd0c78a8776 Oct 31 03:59:02.280456 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:02.280421 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" event={"ID":"bca09c97-0fd2-490b-8a4b-ef50aa274650","Type":"ContainerStarted","Data":"5ddd54264198f2a0d0f034b3453416fc127f19888a8f85671c503cd0c78a8776"} Oct 31 03:59:02.367706 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:02.367593 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:03.287350 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.287251 4488 generic.go:356] "Generic (PLEG): container finished" podID="bca09c97-0fd2-490b-8a4b-ef50aa274650" containerID="ebe63de2ea5f158835ff81cc297e3781756178f6e5ae720a2e84efce0ff47846" exitCode=0 Oct 31 03:59:03.287754 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.287337 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" event={"ID":"bca09c97-0fd2-490b-8a4b-ef50aa274650","Type":"ContainerDied","Data":"ebe63de2ea5f158835ff81cc297e3781756178f6e5ae720a2e84efce0ff47846"} Oct 31 03:59:03.562661 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.562569 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" containerID="cri-o://4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5" gracePeriod=45 Oct 31 03:59:03.652613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.652567 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused" Oct 31 03:59:03.831806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.831781 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:59:03.906002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.905969 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home\") pod \"fb5ef745-c36e-4038-b875-7940030f51d2\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " Oct 31 03:59:03.906002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906015 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs\") pod \"fb5ef745-c36e-4038-b875-7940030f51d2\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " Oct 31 03:59:03.906512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906084 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir\") pod \"fb5ef745-c36e-4038-b875-7940030f51d2\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " Oct 31 03:59:03.906512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906189 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm\") pod \"fb5ef745-c36e-4038-b875-7940030f51d2\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " Oct 31 03:59:03.906512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906224 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5\" (UniqueName: \"kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5\") pod \"fb5ef745-c36e-4038-b875-7940030f51d2\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " Oct 31 03:59:03.906512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906307 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache\") pod \"fb5ef745-c36e-4038-b875-7940030f51d2\" (UID: \"fb5ef745-c36e-4038-b875-7940030f51d2\") " Oct 31 03:59:03.906512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906323 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home" pod "fb5ef745-c36e-4038-b875-7940030f51d2" (UID: "fb5ef745-c36e-4038-b875-7940030f51d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:03.906512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906385 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir" pod "fb5ef745-c36e-4038-b875-7940030f51d2" (UID: "fb5ef745-c36e-4038-b875-7940030f51d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:03.906819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906587 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache" pod "fb5ef745-c36e-4038-b875-7940030f51d2" (UID: "fb5ef745-c36e-4038-b875-7940030f51d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:03.906819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906667 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:03.906819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906683 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:03.906819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.906694 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:03.908565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.908534 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs" pod "fb5ef745-c36e-4038-b875-7940030f51d2" (UID: "fb5ef745-c36e-4038-b875-7940030f51d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:03.908684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.908611 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5" pod "fb5ef745-c36e-4038-b875-7940030f51d2" (UID: "fb5ef745-c36e-4038-b875-7940030f51d2"). InnerVolumeSpecName "kube-api-access-txmk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:03.908684 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:03.908619 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm" pod "fb5ef745-c36e-4038-b875-7940030f51d2" (UID: "fb5ef745-c36e-4038-b875-7940030f51d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:04.007345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.007310 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb5ef745-c36e-4038-b875-7940030f51d2-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.007345 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.007343 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb5ef745-c36e-4038-b875-7940030f51d2-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.007651 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.007356 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txmk5\" (UniqueName: \"kubernetes.io/projected/fb5ef745-c36e-4038-b875-7940030f51d2-kube-api-access-txmk5\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.293949 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.293915 4488 generic.go:356] "Generic (PLEG): container finished" podID="fb5ef745-c36e-4038-b875-7940030f51d2" containerID="4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5" exitCode=0 Oct 31 03:59:04.294505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.294006 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" event={"ID":"fb5ef745-c36e-4038-b875-7940030f51d2","Type":"ContainerDied","Data":"4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5"} Oct 31 03:59:04.294505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.294052 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" event={"ID":"fb5ef745-c36e-4038-b875-7940030f51d2","Type":"ContainerDied","Data":"efacb776976fca35f011f0176a7b2951ba0cd305eac47323f4d1097dbfef7d46"} Oct 31 03:59:04.294505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.294071 4488 scope.go:122] "RemoveContainer" containerID="4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5" Oct 31 03:59:04.294505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.294200 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf" Oct 31 03:59:04.305777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.305755 4488 scope.go:122] "RemoveContainer" containerID="4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5" Oct 31 03:59:04.306140 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:04.306121 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5\": container with ID starting with 4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5 not found: ID does not exist" containerID="4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5" Oct 31 03:59:04.306185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.306150 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5"} err="failed to get container status \"4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5\": rpc error: code = NotFound desc = could not find container \"4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5\": container with ID starting with 4b58ca8df8580ffe4f799345b5f4d03bcd28e01cf46d0f5b25da428d409704b5 not found: ID does not exist" Oct 31 03:59:04.436441 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.436411 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:04.462793 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.462754 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf"] Oct 31 03:59:04.466675 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.466645 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b98893/premium-w3-b98893-kserve-57648f8784-4qqlf"] Oct 31 03:59:04.513737 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.513696 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca\") pod \"bca09c97-0fd2-490b-8a4b-ef50aa274650\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " Oct 31 03:59:04.513918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.513743 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6\" (UniqueName: \"kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6\") pod \"bca09c97-0fd2-490b-8a4b-ef50aa274650\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " Oct 31 03:59:04.513918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.513825 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca\") pod \"bca09c97-0fd2-490b-8a4b-ef50aa274650\" (UID: \"bca09c97-0fd2-490b-8a4b-ef50aa274650\") " Oct 31 03:59:04.514259 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.514231 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca" pod "bca09c97-0fd2-490b-8a4b-ef50aa274650" (UID: "bca09c97-0fd2-490b-8a4b-ef50aa274650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:04.515991 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.515962 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca" pod "bca09c97-0fd2-490b-8a4b-ef50aa274650" (UID: "bca09c97-0fd2-490b-8a4b-ef50aa274650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:04.516140 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.516117 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6" pod "bca09c97-0fd2-490b-8a4b-ef50aa274650" (UID: "bca09c97-0fd2-490b-8a4b-ef50aa274650"). InnerVolumeSpecName "kube-api-access-nlvz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:04.615295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.615265 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvz6\" (UniqueName: \"kubernetes.io/projected/bca09c97-0fd2-490b-8a4b-ef50aa274650-kube-api-access-nlvz6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.615295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.615290 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:04.615295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:04.615303 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca09c97-0fd2-490b-8a4b-ef50aa274650-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:05.301227 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:05.301196 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" Oct 31 03:59:05.301694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:05.301226 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc" event={"ID":"bca09c97-0fd2-490b-8a4b-ef50aa274650","Type":"ContainerDied","Data":"5ddd54264198f2a0d0f034b3453416fc127f19888a8f85671c503cd0c78a8776"} Oct 31 03:59:05.301694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:05.301283 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ddd54264198f2a0d0f034b3453416fc127f19888a8f85671c503cd0c78a8776" Oct 31 03:59:06.316672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:06.316637 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" path="/var/lib/kubelet/pods/fb5ef745-c36e-4038-b875-7940030f51d2/volumes" Oct 31 03:59:06.588954 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:06.588912 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:59:07.204619 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:07.204584 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:08.681481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:08.681447 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:08.839242 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:08.839210 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:09.038193 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.038082 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:09.514461 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.514431 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:09.985867 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.985832 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph"] Oct 31 03:59:09.986407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.986335 4488 cpu_manager.go:414] "containerMap: removing container" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" Oct 31 03:59:09.986407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.986349 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" Oct 31 03:59:09.986407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.986371 4488 cpu_manager.go:414] "containerMap: removing container" podUID="bca09c97-0fd2-490b-8a4b-ef50aa274650" containerName="revoke-keys" Oct 31 03:59:09.986407 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.986376 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca09c97-0fd2-490b-8a4b-ef50aa274650" containerName="revoke-keys" Oct 31 03:59:09.986663 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.986444 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5ef745-c36e-4038-b875-7940030f51d2" containerName="main" Oct 31 03:59:09.986663 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.986455 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca09c97-0fd2-490b-8a4b-ef50aa274650" containerName="revoke-keys" Oct 31 03:59:09.990591 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:09.990571 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.045165 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.045112 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:10.086235 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.086185 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k6s\" (UniqueName: \"kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.086427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.086353 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.086427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.086405 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.095074 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.095046 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:59:10.097548 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.097524 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph"] Oct 31 03:59:10.187940 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.187877 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k6s\" (UniqueName: \"kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.188219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.188008 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.188219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.188048 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.188770 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.188747 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.190682 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.190660 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.249645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.249563 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k6s\" (UniqueName: \"kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s\") pod \"maas-api-revoke-keys-e2e-worker-w0-909365-c6lph\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.304834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.304798 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:10.685720 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:10.685600 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph"] Oct 31 03:59:10.689317 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:10.689283 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80d7104_4cb2_4383_850a_13d975cd339f.slice/crio-bdf40cf25b211a5b56bb114b3e5d2a021d81af6438380ca8c0048f4775e35e6e WatchSource:0}: Error finding container bdf40cf25b211a5b56bb114b3e5d2a021d81af6438380ca8c0048f4775e35e6e: Status 404 returned error can't find the container with id bdf40cf25b211a5b56bb114b3e5d2a021d81af6438380ca8c0048f4775e35e6e Oct 31 03:59:11.337728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:11.337695 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" event={"ID":"a80d7104-4cb2-4383-850a-13d975cd339f","Type":"ContainerStarted","Data":"b9c602a755da1b3e56eae483ba25bc75e39eba12deed53b70a0a86c4131460a0"} Oct 31 03:59:11.338122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:11.337744 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" event={"ID":"a80d7104-4cb2-4383-850a-13d975cd339f","Type":"ContainerStarted","Data":"bdf40cf25b211a5b56bb114b3e5d2a021d81af6438380ca8c0048f4775e35e6e"} Oct 31 03:59:12.345718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:12.345684 4488 generic.go:356] "Generic (PLEG): container finished" podID="a80d7104-4cb2-4383-850a-13d975cd339f" containerID="b9c602a755da1b3e56eae483ba25bc75e39eba12deed53b70a0a86c4131460a0" exitCode=0 Oct 31 03:59:12.346192 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:12.345777 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" event={"ID":"a80d7104-4cb2-4383-850a-13d975cd339f","Type":"ContainerDied","Data":"b9c602a755da1b3e56eae483ba25bc75e39eba12deed53b70a0a86c4131460a0"} Oct 31 03:59:12.506423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:12.506387 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:12.640773 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:12.640694 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:59:13.586427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.586400 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:13.627578 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.627455 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca\") pod \"a80d7104-4cb2-4383-850a-13d975cd339f\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " Oct 31 03:59:13.627578 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.627548 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s\" (UniqueName: \"kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s\") pod \"a80d7104-4cb2-4383-850a-13d975cd339f\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " Oct 31 03:59:13.627780 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.627592 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca\") pod \"a80d7104-4cb2-4383-850a-13d975cd339f\" (UID: \"a80d7104-4cb2-4383-850a-13d975cd339f\") " Oct 31 03:59:13.628645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.628611 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca" pod "a80d7104-4cb2-4383-850a-13d975cd339f" (UID: "a80d7104-4cb2-4383-850a-13d975cd339f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:13.637294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.637230 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s" pod "a80d7104-4cb2-4383-850a-13d975cd339f" (UID: "a80d7104-4cb2-4383-850a-13d975cd339f"). InnerVolumeSpecName "kube-api-access-d7k6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:13.637447 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.637355 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca" pod "a80d7104-4cb2-4383-850a-13d975cd339f" (UID: "a80d7104-4cb2-4383-850a-13d975cd339f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:13.736422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.736316 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:13.736422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.736359 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k6s\" (UniqueName: \"kubernetes.io/projected/a80d7104-4cb2-4383-850a-13d975cd339f-kube-api-access-d7k6s\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:13.736422 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:13.736378 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80d7104-4cb2-4383-850a-13d975cd339f-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:14.378175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:14.377632 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" event={"ID":"a80d7104-4cb2-4383-850a-13d975cd339f","Type":"ContainerDied","Data":"bdf40cf25b211a5b56bb114b3e5d2a021d81af6438380ca8c0048f4775e35e6e"} Oct 31 03:59:14.378175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:14.377698 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf40cf25b211a5b56bb114b3e5d2a021d81af6438380ca8c0048f4775e35e6e" Oct 31 03:59:14.378175 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:14.377771 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph" Oct 31 03:59:14.511134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:14.502543 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv"] Oct 31 03:59:14.514419 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:14.514376 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7"] Oct 31 03:59:14.594874 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:14.593828 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:15.606061 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:15.606021 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 03:59:15.778577 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:15.777232 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d" podUID="e252476b-aafe-4d02-9673-c55bbb915a52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:59:17.809247 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:17.809196 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:18.873314 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:18.873276 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:19.628112 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:19.627381 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:20.397038 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:20.396992 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:20.564281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:20.564238 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:21.601798 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:21.601764 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2"] Oct 31 03:59:21.875327 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:21.875234 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:22.042515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.042469 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:22.071752 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.071715 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65"] Oct 31 03:59:22.072064 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.072039 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" podUID="c620b32c-0e8a-4118-92f9-275b62e09266" containerName="maas-api" containerID="cri-o://0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e" gracePeriod=30 Oct 31 03:59:22.205316 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.205278 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:22.392836 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.392807 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:59:22.506945 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.506725 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6\" (UniqueName: \"kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6\") pod \"c620b32c-0e8a-4118-92f9-275b62e09266\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.509790 4488 generic.go:356] "Generic (PLEG): container finished" podID="c620b32c-0e8a-4118-92f9-275b62e09266" containerID="0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e" exitCode=0 Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.509952 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" event={"ID":"c620b32c-0e8a-4118-92f9-275b62e09266","Type":"ContainerDied","Data":"0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e"} Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.510010 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" event={"ID":"c620b32c-0e8a-4118-92f9-275b62e09266","Type":"ContainerDied","Data":"b816ed7723e016d84f4fdc2c5429892839fa29e012f76b9ef08b0a3f2179e80e"} Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.510043 4488 scope.go:122] "RemoveContainer" containerID="0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e" Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.510248 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65" Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.512364 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca\") pod \"c620b32c-0e8a-4118-92f9-275b62e09266\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.512423 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\") pod \"c620b32c-0e8a-4118-92f9-275b62e09266\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.512501 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs\") pod \"c620b32c-0e8a-4118-92f9-275b62e09266\" (UID: \"c620b32c-0e8a-4118-92f9-275b62e09266\") " Oct 31 03:59:22.517415 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.516392 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca" pod "c620b32c-0e8a-4118-92f9-275b62e09266" (UID: "c620b32c-0e8a-4118-92f9-275b62e09266"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:22.523546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.523437 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs" pod "c620b32c-0e8a-4118-92f9-275b62e09266" (UID: "c620b32c-0e8a-4118-92f9-275b62e09266"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:22.523546 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.523482 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6" pod "c620b32c-0e8a-4118-92f9-275b62e09266" (UID: "c620b32c-0e8a-4118-92f9-275b62e09266"). InnerVolumeSpecName "kube-api-access-9rzm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:22.528340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.526646 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls" pod "c620b32c-0e8a-4118-92f9-275b62e09266" (UID: "c620b32c-0e8a-4118-92f9-275b62e09266"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:22.547168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.547136 4488 scope.go:122] "RemoveContainer" containerID="0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e" Oct 31 03:59:22.547528 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:22.547497 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e\": container with ID starting with 0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e not found: ID does not exist" containerID="0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e" Oct 31 03:59:22.547607 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.547543 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e"} err="failed to get container status \"0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e\": rpc error: code = NotFound desc = could not find container \"0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e\": container with ID starting with 0fc238b550dc70b1e7abba3ee9522378db56028440f39cfec0934fe994aeee0e not found: ID does not exist" Oct 31 03:59:22.614693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.614655 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzm6\" (UniqueName: \"kubernetes.io/projected/c620b32c-0e8a-4118-92f9-275b62e09266-kube-api-access-9rzm6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:22.614693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.614694 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c620b32c-0e8a-4118-92f9-275b62e09266-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:22.614693 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.614710 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:22.615349 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.614723 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c620b32c-0e8a-4118-92f9-275b62e09266-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:22.943596 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.943560 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65"] Oct 31 03:59:22.994930 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:22.994892 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b16de9-597b896f9-g5z65"] Oct 31 03:59:23.750550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:23.750515 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:24.088223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:24.086745 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:24.240588 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:24.240554 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:24.291250 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:24.291198 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz"] Oct 31 03:59:24.318328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:24.318295 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c620b32c-0e8a-4118-92f9-275b62e09266" path="/var/lib/kubelet/pods/c620b32c-0e8a-4118-92f9-275b62e09266/volumes" Oct 31 03:59:24.341122 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:24.341030 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b16de9-phbgz"] Oct 31 03:59:24.716743 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:24.716647 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:26.282339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:26.282309 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:26.318240 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:26.318202 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e23fef4-81c9-4aea-80ef-372890aa8f38" path="/var/lib/kubelet/pods/7e23fef4-81c9-4aea-80ef-372890aa8f38/volumes" Oct 31 03:59:26.441245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:26.441215 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:26.603430 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:26.603386 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerName="payload-pre-processing" containerID="cri-o://fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7" gracePeriod=25 Oct 31 03:59:27.202837 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:27.202802 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:27.460488 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:27.460386 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:27.619616 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:27.619587 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:28.523283 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523253 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m"] Oct 31 03:59:28.523680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523660 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a80d7104-4cb2-4383-850a-13d975cd339f" containerName="revoke-keys" Oct 31 03:59:28.523680 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523671 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80d7104-4cb2-4383-850a-13d975cd339f" containerName="revoke-keys" Oct 31 03:59:28.523774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523687 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c620b32c-0e8a-4118-92f9-275b62e09266" containerName="maas-api" Oct 31 03:59:28.523774 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523694 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c620b32c-0e8a-4118-92f9-275b62e09266" containerName="maas-api" Oct 31 03:59:28.523843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523787 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80d7104-4cb2-4383-850a-13d975cd339f" containerName="revoke-keys" Oct 31 03:59:28.523843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.523795 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c620b32c-0e8a-4118-92f9-275b62e09266" containerName="maas-api" Oct 31 03:59:28.526878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.526863 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.577473 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.577440 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v"] Oct 31 03:59:28.580894 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.580874 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.588657 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.588624 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.588843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.588697 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.588912 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.588836 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnsz\" (UniqueName: \"kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.623988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.623937 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:28.624239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.623999 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:28.624239 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.624023 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:28.630046 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.630024 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:59:28.632575 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.632550 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m"] Oct 31 03:59:28.638173 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.638141 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v"] Oct 31 03:59:28.690110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.690070 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfsw\" (UniqueName: \"kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.690287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.690146 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.690287 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.690269 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.690374 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.690320 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.690408 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.690374 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.690450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.690407 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnsz\" (UniqueName: \"kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.691105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.691065 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.692875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.692853 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.751558 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.751520 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnsz\" (UniqueName: \"kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz\") pod \"maas-api-revoke-sub-74787b76-4rn8m\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.791459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.791368 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.791459 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.791429 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfsw\" (UniqueName: \"kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.791658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.791508 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.792170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.792140 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.793848 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.793828 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.839932 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.839893 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:28.852843 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.852815 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfsw\" (UniqueName: \"kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw\") pod \"maas-api-revoke-sub-0a8432d7-ntq7v\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:28.899472 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:28.899441 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:29.217434 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.217410 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m"] Oct 31 03:59:29.219037 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:29.219008 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd291a63e_2d80_4613_bfaa_2ee7e254d19a.slice/crio-5e1bbd20fbc9bb7e353e8c5dba6db352c11dafe559feb3777270722b05de7bbe WatchSource:0}: Error finding container 5e1bbd20fbc9bb7e353e8c5dba6db352c11dafe559feb3777270722b05de7bbe: Status 404 returned error can't find the container with id 5e1bbd20fbc9bb7e353e8c5dba6db352c11dafe559feb3777270722b05de7bbe Oct 31 03:59:29.328804 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.328774 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v"] Oct 31 03:59:29.330529 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:29.330501 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf317710a_ecc1_456f_adc7_106258083322.slice/crio-a9b1b55df3ad797fd0fc7112dffa47d0a4e028a985076aee76ed941e7b7bf134 WatchSource:0}: Error finding container a9b1b55df3ad797fd0fc7112dffa47d0a4e028a985076aee76ed941e7b7bf134: Status 404 returned error can't find the container with id a9b1b55df3ad797fd0fc7112dffa47d0a4e028a985076aee76ed941e7b7bf134 Oct 31 03:59:29.437166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.437138 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:29.553245 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.553211 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" event={"ID":"f317710a-ecc1-456f-adc7-106258083322","Type":"ContainerStarted","Data":"a9b1b55df3ad797fd0fc7112dffa47d0a4e028a985076aee76ed941e7b7bf134"} Oct 31 03:59:29.554691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.554669 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" event={"ID":"d291a63e-2d80-4613-bfaa-2ee7e254d19a","Type":"ContainerStarted","Data":"f7ad3ce333eb69fc71ced6f8d903f22e2eee95134c7434859418cd432893459b"} Oct 31 03:59:29.554819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.554713 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" event={"ID":"d291a63e-2d80-4613-bfaa-2ee7e254d19a","Type":"ContainerStarted","Data":"5e1bbd20fbc9bb7e353e8c5dba6db352c11dafe559feb3777270722b05de7bbe"} Oct 31 03:59:29.673256 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:29.673203 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" podStartSLOduration=1.6731856980000002 podStartE2EDuration="1.673185698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:29.67187102 +0000 UTC m=+2397.468589278" watchObservedRunningTime="2026-10-31 03:59:29.673185698 +0000 UTC m=+2397.469903958" Oct 31 03:59:30.561321 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:30.561226 4488 generic.go:356] "Generic (PLEG): container finished" podID="f317710a-ecc1-456f-adc7-106258083322" containerID="925ade31a42e7a409d7c0f0635d9e0075146a2b942d793d49ce59ac88faf3343" exitCode=0 Oct 31 03:59:30.561790 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:30.561312 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" event={"ID":"f317710a-ecc1-456f-adc7-106258083322","Type":"ContainerDied","Data":"925ade31a42e7a409d7c0f0635d9e0075146a2b942d793d49ce59ac88faf3343"} Oct 31 03:59:30.562859 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:30.562837 4488 generic.go:356] "Generic (PLEG): container finished" podID="d291a63e-2d80-4613-bfaa-2ee7e254d19a" containerID="f7ad3ce333eb69fc71ced6f8d903f22e2eee95134c7434859418cd432893459b" exitCode=0 Oct 31 03:59:30.562979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:30.562869 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" event={"ID":"d291a63e-2d80-4613-bfaa-2ee7e254d19a","Type":"ContainerDied","Data":"f7ad3ce333eb69fc71ced6f8d903f22e2eee95134c7434859418cd432893459b"} Oct 31 03:59:30.713725 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:30.713694 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:30.800079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:30.800044 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" containerID="cri-o://e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735" gracePeriod=44 Oct 31 03:59:31.070718 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.070619 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" containerID="cri-o://08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e" gracePeriod=44 Oct 31 03:59:31.077782 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.077764 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:59:31.221339 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221306 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home\") pod \"65ee8d7e-e337-4970-862f-9da26774ccbb\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " Oct 31 03:59:31.221537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221417 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs\") pod \"65ee8d7e-e337-4970-862f-9da26774ccbb\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " Oct 31 03:59:31.221537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221484 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache\") pod \"65ee8d7e-e337-4970-862f-9da26774ccbb\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " Oct 31 03:59:31.221672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221596 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir\") pod \"65ee8d7e-e337-4970-862f-9da26774ccbb\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " Oct 31 03:59:31.221672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221630 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm\") pod \"65ee8d7e-e337-4970-862f-9da26774ccbb\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " Oct 31 03:59:31.221672 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221634 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home" pod "65ee8d7e-e337-4970-862f-9da26774ccbb" (UID: "65ee8d7e-e337-4970-862f-9da26774ccbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.221833 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.221735 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t\" (UniqueName: \"kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t\") pod \"65ee8d7e-e337-4970-862f-9da26774ccbb\" (UID: \"65ee8d7e-e337-4970-862f-9da26774ccbb\") " Oct 31 03:59:31.222180 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.222030 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir" pod "65ee8d7e-e337-4970-862f-9da26774ccbb" (UID: "65ee8d7e-e337-4970-862f-9da26774ccbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.222301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.222184 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.222301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.222202 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.222301 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.222258 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache" pod "65ee8d7e-e337-4970-862f-9da26774ccbb" (UID: "65ee8d7e-e337-4970-862f-9da26774ccbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.224427 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.224402 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t" pod "65ee8d7e-e337-4970-862f-9da26774ccbb" (UID: "65ee8d7e-e337-4970-862f-9da26774ccbb"). InnerVolumeSpecName "kube-api-access-mcc8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:31.228681 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.228650 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm" pod "65ee8d7e-e337-4970-862f-9da26774ccbb" (UID: "65ee8d7e-e337-4970-862f-9da26774ccbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.230671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.230635 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs" pod "65ee8d7e-e337-4970-862f-9da26774ccbb" (UID: "65ee8d7e-e337-4970-862f-9da26774ccbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:31.323474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.323385 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee8d7e-e337-4970-862f-9da26774ccbb-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.323474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.323416 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.323474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.323426 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee8d7e-e337-4970-862f-9da26774ccbb-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.323474 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.323435 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcc8t\" (UniqueName: \"kubernetes.io/projected/65ee8d7e-e337-4970-862f-9da26774ccbb-kube-api-access-mcc8t\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.356839 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.356802 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:59:31.424322 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424287 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm\") pod \"219920c9-66f5-48e8-87af-011113d40627\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " Oct 31 03:59:31.424505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424391 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache\") pod \"219920c9-66f5-48e8-87af-011113d40627\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " Oct 31 03:59:31.424505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424434 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l\" (UniqueName: \"kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l\") pod \"219920c9-66f5-48e8-87af-011113d40627\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " Oct 31 03:59:31.424505 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424490 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home\") pod \"219920c9-66f5-48e8-87af-011113d40627\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " Oct 31 03:59:31.424666 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424514 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs\" (UniqueName: \"kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs\") pod \"219920c9-66f5-48e8-87af-011113d40627\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " Oct 31 03:59:31.424666 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424583 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir\") pod \"219920c9-66f5-48e8-87af-011113d40627\" (UID: \"219920c9-66f5-48e8-87af-011113d40627\") " Oct 31 03:59:31.424764 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424708 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache" pod "219920c9-66f5-48e8-87af-011113d40627" (UID: "219920c9-66f5-48e8-87af-011113d40627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.424828 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424780 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home" pod "219920c9-66f5-48e8-87af-011113d40627" (UID: "219920c9-66f5-48e8-87af-011113d40627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.424979 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.424947 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir" pod "219920c9-66f5-48e8-87af-011113d40627" (UID: "219920c9-66f5-48e8-87af-011113d40627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.425225 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.425204 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.425378 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.425265 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.425426 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.425385 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.426801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.426779 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm" pod "219920c9-66f5-48e8-87af-011113d40627" (UID: "219920c9-66f5-48e8-87af-011113d40627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:31.426875 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.426795 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs" pod "219920c9-66f5-48e8-87af-011113d40627" (UID: "219920c9-66f5-48e8-87af-011113d40627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:31.426927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.426893 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l" pod "219920c9-66f5-48e8-87af-011113d40627" (UID: "219920c9-66f5-48e8-87af-011113d40627"). InnerVolumeSpecName "kube-api-access-h549l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:31.526551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.526517 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h549l\" (UniqueName: \"kubernetes.io/projected/219920c9-66f5-48e8-87af-011113d40627-kube-api-access-h549l\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.526551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.526548 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219920c9-66f5-48e8-87af-011113d40627-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.526551 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.526560 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219920c9-66f5-48e8-87af-011113d40627-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.571252 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.571222 4488 generic.go:356] "Generic (PLEG): container finished" podID="219920c9-66f5-48e8-87af-011113d40627" containerID="08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e" exitCode=0 Oct 31 03:59:31.571766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.571291 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" Oct 31 03:59:31.571766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.571298 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" event={"ID":"219920c9-66f5-48e8-87af-011113d40627","Type":"ContainerDied","Data":"08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e"} Oct 31 03:59:31.571766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.571341 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7" event={"ID":"219920c9-66f5-48e8-87af-011113d40627","Type":"ContainerDied","Data":"9a17c22f2c536bf7239929c749d11d14ed3f748dacb6507ed19d385c4908c6ea"} Oct 31 03:59:31.571766 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.571358 4488 scope.go:122] "RemoveContainer" containerID="08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e" Oct 31 03:59:31.573280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.573188 4488 generic.go:356] "Generic (PLEG): container finished" podID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerID="e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735" exitCode=0 Oct 31 03:59:31.573390 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.573233 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" Oct 31 03:59:31.573581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.573250 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" event={"ID":"65ee8d7e-e337-4970-862f-9da26774ccbb","Type":"ContainerDied","Data":"e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735"} Oct 31 03:59:31.573581 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.573509 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv" event={"ID":"65ee8d7e-e337-4970-862f-9da26774ccbb","Type":"ContainerDied","Data":"79d1a2725cc5fc2c602b9266eb5a3b52ebe4caf96d2263e050eb2c927fc45c11"} Oct 31 03:59:31.584554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.584206 4488 scope.go:122] "RemoveContainer" containerID="08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e" Oct 31 03:59:31.584719 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:31.584660 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e\": container with ID starting with 08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e not found: ID does not exist" containerID="08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e" Oct 31 03:59:31.584854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.584718 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e"} err="failed to get container status \"08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e\": rpc error: code = NotFound desc = could not find container \"08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e\": container with ID starting with 08142920c16e023102e11bdce632842122687d7bd5cacdd16bbb278541c2893e not found: ID does not exist" Oct 31 03:59:31.584854 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.584739 4488 scope.go:122] "RemoveContainer" containerID="e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735" Oct 31 03:59:31.599985 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.599943 4488 scope.go:122] "RemoveContainer" containerID="e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735" Oct 31 03:59:31.600466 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:31.600440 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735\": container with ID starting with e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735 not found: ID does not exist" containerID="e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735" Oct 31 03:59:31.600568 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.600481 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735"} err="failed to get container status \"e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735\": rpc error: code = NotFound desc = could not find container \"e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735\": container with ID starting with e6719f7c83f15ff956e273e96b839287fafc05c4c067ae282ff2c47006985735 not found: ID does not exist" Oct 31 03:59:31.726946 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.726925 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:31.733512 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.733483 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:31.735810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.735782 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv"] Oct 31 03:59:31.742543 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.742516 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909365/premium-w0-909365-kserve-74b7d98cc8-72gfv"] Oct 31 03:59:31.829423 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829397 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw\" (UniqueName: \"kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw\") pod \"f317710a-ecc1-456f-adc7-106258083322\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " Oct 31 03:59:31.829574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829461 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca\") pod \"f317710a-ecc1-456f-adc7-106258083322\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " Oct 31 03:59:31.829574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829505 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz\" (UniqueName: \"kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz\") pod \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " Oct 31 03:59:31.829574 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829546 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca\") pod \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " Oct 31 03:59:31.829708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829635 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca\") pod \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\" (UID: \"d291a63e-2d80-4613-bfaa-2ee7e254d19a\") " Oct 31 03:59:31.829708 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829693 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca\") pod \"f317710a-ecc1-456f-adc7-106258083322\" (UID: \"f317710a-ecc1-456f-adc7-106258083322\") " Oct 31 03:59:31.829876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829847 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca" pod "f317710a-ecc1-456f-adc7-106258083322" (UID: "f317710a-ecc1-456f-adc7-106258083322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:31.829961 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.829902 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca" pod "d291a63e-2d80-4613-bfaa-2ee7e254d19a" (UID: "d291a63e-2d80-4613-bfaa-2ee7e254d19a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:31.830228 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.830203 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f317710a-ecc1-456f-adc7-106258083322-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.830360 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.830232 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.831983 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.831941 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca" pod "d291a63e-2d80-4613-bfaa-2ee7e254d19a" (UID: "d291a63e-2d80-4613-bfaa-2ee7e254d19a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:31.832082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.831993 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca" pod "f317710a-ecc1-456f-adc7-106258083322" (UID: "f317710a-ecc1-456f-adc7-106258083322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:31.832082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.832016 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz" pod "d291a63e-2d80-4613-bfaa-2ee7e254d19a" (UID: "d291a63e-2d80-4613-bfaa-2ee7e254d19a"). InnerVolumeSpecName "kube-api-access-flnsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:31.832082 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.832062 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw" pod "f317710a-ecc1-456f-adc7-106258083322" (UID: "f317710a-ecc1-456f-adc7-106258083322"). InnerVolumeSpecName "kube-api-access-xgfsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:31.851918 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.851876 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7"] Oct 31 03:59:31.856311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.856283 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909365/simulator-w0-909365-kserve-58648bbd79-xbvz7"] Oct 31 03:59:31.931157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.931119 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnsz\" (UniqueName: \"kubernetes.io/projected/d291a63e-2d80-4613-bfaa-2ee7e254d19a-kube-api-access-flnsz\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.931157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.931153 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d291a63e-2d80-4613-bfaa-2ee7e254d19a-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.931157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.931165 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f317710a-ecc1-456f-adc7-106258083322-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:31.931440 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:31.931176 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfsw\" (UniqueName: \"kubernetes.io/projected/f317710a-ecc1-456f-adc7-106258083322-kube-api-access-xgfsw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.317032 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.316949 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219920c9-66f5-48e8-87af-011113d40627" path="/var/lib/kubelet/pods/219920c9-66f5-48e8-87af-011113d40627/volumes" Oct 31 03:59:32.317401 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.317385 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" path="/var/lib/kubelet/pods/65ee8d7e-e337-4970-862f-9da26774ccbb/volumes" Oct 31 03:59:32.381627 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.381594 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n"] Oct 31 03:59:32.382150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382131 4488 cpu_manager.go:414] "containerMap: removing container" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" Oct 31 03:59:32.382150 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382151 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" Oct 31 03:59:32.382329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382165 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d291a63e-2d80-4613-bfaa-2ee7e254d19a" containerName="revoke-keys" Oct 31 03:59:32.382329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382172 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d291a63e-2d80-4613-bfaa-2ee7e254d19a" containerName="revoke-keys" Oct 31 03:59:32.382329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382191 4488 cpu_manager.go:414] "containerMap: removing container" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" Oct 31 03:59:32.382329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382197 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" Oct 31 03:59:32.382329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382217 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f317710a-ecc1-456f-adc7-106258083322" containerName="revoke-keys" Oct 31 03:59:32.382329 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382224 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f317710a-ecc1-456f-adc7-106258083322" containerName="revoke-keys" Oct 31 03:59:32.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382356 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ee8d7e-e337-4970-862f-9da26774ccbb" containerName="main" Oct 31 03:59:32.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382370 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f317710a-ecc1-456f-adc7-106258083322" containerName="revoke-keys" Oct 31 03:59:32.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382389 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d291a63e-2d80-4613-bfaa-2ee7e254d19a" containerName="revoke-keys" Oct 31 03:59:32.382628 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.382398 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="219920c9-66f5-48e8-87af-011113d40627" containerName="main" Oct 31 03:59:32.386594 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.386568 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.443037 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.442990 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n"] Oct 31 03:59:32.444801 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.444768 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:32.537862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.537819 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw5c\" (UniqueName: \"kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.537862 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.537866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.538154 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.537888 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.581614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.581585 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" Oct 31 03:59:32.581614 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.581603 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m" event={"ID":"d291a63e-2d80-4613-bfaa-2ee7e254d19a","Type":"ContainerDied","Data":"5e1bbd20fbc9bb7e353e8c5dba6db352c11dafe559feb3777270722b05de7bbe"} Oct 31 03:59:32.582126 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.581655 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e1bbd20fbc9bb7e353e8c5dba6db352c11dafe559feb3777270722b05de7bbe" Oct 31 03:59:32.584246 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.584227 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" Oct 31 03:59:32.584398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.584264 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v" event={"ID":"f317710a-ecc1-456f-adc7-106258083322","Type":"ContainerDied","Data":"a9b1b55df3ad797fd0fc7112dffa47d0a4e028a985076aee76ed941e7b7bf134"} Oct 31 03:59:32.584398 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.584312 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b1b55df3ad797fd0fc7112dffa47d0a4e028a985076aee76ed941e7b7bf134" Oct 31 03:59:32.639047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.639009 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw5c\" (UniqueName: \"kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.639047 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.639055 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.639353 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.639199 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.639707 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.639684 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.641882 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.641861 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.700018 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.699984 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw5c\" (UniqueName: \"kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c\") pod \"maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:32.700750 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:32.700729 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:33.086638 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:33.086610 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n"] Oct 31 03:59:33.088905 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:33.088876 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee203307_1664_49e3_ae66_bad5d5c06167.slice/crio-4f2dd5b2fee0c3fd80a5ea6edba2f5f25b9207b5fa2326bf16bc904e2d29d7eb WatchSource:0}: Error finding container 4f2dd5b2fee0c3fd80a5ea6edba2f5f25b9207b5fa2326bf16bc904e2d29d7eb: Status 404 returned error can't find the container with id 4f2dd5b2fee0c3fd80a5ea6edba2f5f25b9207b5fa2326bf16bc904e2d29d7eb Oct 31 03:59:33.582845 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:33.582799 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:33.592166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:33.592132 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" event={"ID":"ee203307-1664-49e3-ae66-bad5d5c06167","Type":"ContainerStarted","Data":"0d8679153d66ea4e9856d2b6881a792f894127a5fe05f57f638a4f2c231e400a"} Oct 31 03:59:33.592361 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:33.592190 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" event={"ID":"ee203307-1664-49e3-ae66-bad5d5c06167","Type":"ContainerStarted","Data":"4f2dd5b2fee0c3fd80a5ea6edba2f5f25b9207b5fa2326bf16bc904e2d29d7eb"} Oct 31 03:59:33.710335 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:33.710224 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" podStartSLOduration=1.710204931 podStartE2EDuration="1.710204931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:33.708400523 +0000 UTC m=+2401.505118783" watchObservedRunningTime="2026-10-31 03:59:33.710204931 +0000 UTC m=+2401.506923187" Oct 31 03:59:33.945207 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:33.945172 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:34.599554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:34.599517 4488 generic.go:356] "Generic (PLEG): container finished" podID="ee203307-1664-49e3-ae66-bad5d5c06167" containerID="0d8679153d66ea4e9856d2b6881a792f894127a5fe05f57f638a4f2c231e400a" exitCode=0 Oct 31 03:59:34.600033 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:34.599610 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" event={"ID":"ee203307-1664-49e3-ae66-bad5d5c06167","Type":"ContainerDied","Data":"0d8679153d66ea4e9856d2b6881a792f894127a5fe05f57f638a4f2c231e400a"} Oct 31 03:59:35.377890 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.377856 4488 scope.go:122] "RemoveContainer" containerID="8ac712eeedd7092d56c4193936aad95fedd375d8e55f5365b30ecf9fe4f5111a" Oct 31 03:59:35.583810 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.583771 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:35.755779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.755747 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:35.873674 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.873640 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:35.884963 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.884930 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca\") pod \"ee203307-1664-49e3-ae66-bad5d5c06167\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " Oct 31 03:59:35.885185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.884974 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c\" (UniqueName: \"kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c\") pod \"ee203307-1664-49e3-ae66-bad5d5c06167\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " Oct 31 03:59:35.885185 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.885015 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca\") pod \"ee203307-1664-49e3-ae66-bad5d5c06167\" (UID: \"ee203307-1664-49e3-ae66-bad5d5c06167\") " Oct 31 03:59:35.885477 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.885448 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca" pod "ee203307-1664-49e3-ae66-bad5d5c06167" (UID: "ee203307-1664-49e3-ae66-bad5d5c06167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:35.887356 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.887325 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca" pod "ee203307-1664-49e3-ae66-bad5d5c06167" (UID: "ee203307-1664-49e3-ae66-bad5d5c06167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:35.887468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.887365 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c" pod "ee203307-1664-49e3-ae66-bad5d5c06167" (UID: "ee203307-1664-49e3-ae66-bad5d5c06167"). InnerVolumeSpecName "kube-api-access-9hw5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:35.986325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.986231 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:35.986325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.986265 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hw5c\" (UniqueName: \"kubernetes.io/projected/ee203307-1664-49e3-ae66-bad5d5c06167-kube-api-access-9hw5c\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:35.986325 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:35.986277 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee203307-1664-49e3-ae66-bad5d5c06167-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:36.026515 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.026465 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:36.124679 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.124560 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf"] Oct 31 03:59:36.128157 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.128127 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b63425-wx4gf"] Oct 31 03:59:36.315806 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.315725 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0dca2c-863f-4003-9f03-6bdad394b110" path="/var/lib/kubelet/pods/1d0dca2c-863f-4003-9f03-6bdad394b110/volumes" Oct 31 03:59:36.562121 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.562057 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:36.590444 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:36.590412 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 03:59:36.612387 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.612359 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" Oct 31 03:59:36.612569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.612349 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n" event={"ID":"ee203307-1664-49e3-ae66-bad5d5c06167","Type":"ContainerDied","Data":"4f2dd5b2fee0c3fd80a5ea6edba2f5f25b9207b5fa2326bf16bc904e2d29d7eb"} Oct 31 03:59:36.612569 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.612489 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2dd5b2fee0c3fd80a5ea6edba2f5f25b9207b5fa2326bf16bc904e2d29d7eb" Oct 31 03:59:36.858506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.858414 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p"] Oct 31 03:59:36.863102 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:36.863068 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z"] Oct 31 03:59:37.052605 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:37.052568 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:37.202328 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:37.202237 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:40.434267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.434228 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw"] Oct 31 03:59:40.434694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.434678 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ee203307-1664-49e3-ae66-bad5d5c06167" containerName="revoke-keys" Oct 31 03:59:40.434694 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.434689 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee203307-1664-49e3-ae66-bad5d5c06167" containerName="revoke-keys" Oct 31 03:59:40.434779 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.434754 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee203307-1664-49e3-ae66-bad5d5c06167" containerName="revoke-keys" Oct 31 03:59:40.438160 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.438144 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.440311 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.440283 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.440435 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.440328 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdc2\" (UniqueName: \"kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.440537 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.440509 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.485219 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.485188 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts"] Oct 31 03:59:40.488794 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.488771 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.495188 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.495159 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw"] Oct 31 03:59:40.532384 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.532350 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:40.532384 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.532364 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:40.541281 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.541251 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.541436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.541334 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.541436 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.541383 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrsq\" (UniqueName: \"kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.541550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.541435 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.541550 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.541469 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdc2\" (UniqueName: \"kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.541664 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.541542 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.542166 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.542147 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.542285 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.542269 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:40.542373 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.542353 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:59:40.547816 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.547792 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts"] Oct 31 03:59:40.555223 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.555200 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.603013 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.602984 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdc2\" (UniqueName: \"kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2\") pod \"maas-api-revoke-sub-7fdbde72-h62kw\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.642294 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.642261 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.642317 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrsq\" (UniqueName: \"kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.642485 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.642365 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.643002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.642976 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.644802 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.644782 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.703857 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.703780 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrsq\" (UniqueName: \"kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq\") pod \"maas-api-revoke-sub-63184e1b-tgcts\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:40.751878 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.751840 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:40.805073 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:40.805048 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:41.127797 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.127771 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw"] Oct 31 03:59:41.129768 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:41.129743 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534638ae_3b4a_4b36_8171_5030e668df3e.slice/crio-21c3a1ef31436431df9879053edc0a226ad0c52b6f74464ec4d449d9cc59d396 WatchSource:0}: Error finding container 21c3a1ef31436431df9879053edc0a226ad0c52b6f74464ec4d449d9cc59d396: Status 404 returned error can't find the container with id 21c3a1ef31436431df9879053edc0a226ad0c52b6f74464ec4d449d9cc59d396 Oct 31 03:59:41.229645 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.229618 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts"] Oct 31 03:59:41.231617 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:41.231591 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36123b2_5c3f_4f3f_83da_01acc9a4317e.slice/crio-439d9eb3a738db2e08c11c5cb4a3d8acf129548907925a5f5c95b89dde8c25d6 WatchSource:0}: Error finding container 439d9eb3a738db2e08c11c5cb4a3d8acf129548907925a5f5c95b89dde8c25d6: Status 404 returned error can't find the container with id 439d9eb3a738db2e08c11c5cb4a3d8acf129548907925a5f5c95b89dde8c25d6 Oct 31 03:59:41.334785 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.334751 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:41.641143 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.641083 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" event={"ID":"d36123b2-5c3f-4f3f-83da-01acc9a4317e","Type":"ContainerStarted","Data":"d0f37e978ad07c11b564fece391c44c8130d04639a505660cf3df9c3e97f9971"} Oct 31 03:59:41.641563 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.641163 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" event={"ID":"d36123b2-5c3f-4f3f-83da-01acc9a4317e","Type":"ContainerStarted","Data":"439d9eb3a738db2e08c11c5cb4a3d8acf129548907925a5f5c95b89dde8c25d6"} Oct 31 03:59:41.642728 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.642702 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" event={"ID":"534638ae-3b4a-4b36-8171-5030e668df3e","Type":"ContainerStarted","Data":"bbae2834008fec825bbb7f17221ff76616de8af316a1a6fe8acde1769120a59d"} Oct 31 03:59:41.642818 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.642747 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" event={"ID":"534638ae-3b4a-4b36-8171-5030e668df3e","Type":"ContainerStarted","Data":"21c3a1ef31436431df9879053edc0a226ad0c52b6f74464ec4d449d9cc59d396"} Oct 31 03:59:41.758671 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.758563 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" podStartSLOduration=1.7585495230000001 podStartE2EDuration="1.758549523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:41.756242786 +0000 UTC m=+2409.552961038" watchObservedRunningTime="2026-10-31 03:59:41.758549523 +0000 UTC m=+2409.555267781" Oct 31 03:59:41.874078 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:41.874029 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" podStartSLOduration=1.8740142340000001 podStartE2EDuration="1.874014234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:41.871227993 +0000 UTC m=+2409.667946265" watchObservedRunningTime="2026-10-31 03:59:41.874014234 +0000 UTC m=+2409.670732537" Oct 31 03:59:42.648988 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:42.648953 4488 generic.go:356] "Generic (PLEG): container finished" podID="534638ae-3b4a-4b36-8171-5030e668df3e" containerID="bbae2834008fec825bbb7f17221ff76616de8af316a1a6fe8acde1769120a59d" exitCode=0 Oct 31 03:59:42.649502 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:42.649042 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" event={"ID":"534638ae-3b4a-4b36-8171-5030e668df3e","Type":"ContainerDied","Data":"bbae2834008fec825bbb7f17221ff76616de8af316a1a6fe8acde1769120a59d"} Oct 31 03:59:42.650658 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:42.650636 4488 generic.go:356] "Generic (PLEG): container finished" podID="d36123b2-5c3f-4f3f-83da-01acc9a4317e" containerID="d0f37e978ad07c11b564fece391c44c8130d04639a505660cf3df9c3e97f9971" exitCode=0 Oct 31 03:59:42.650760 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:42.650667 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" event={"ID":"d36123b2-5c3f-4f3f-83da-01acc9a4317e","Type":"ContainerDied","Data":"d0f37e978ad07c11b564fece391c44c8130d04639a505660cf3df9c3e97f9971"} Oct 31 03:59:42.726777 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:42.726745 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:42.938879 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:42.938784 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:43.164065 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.164030 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:43.808371 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.808337 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:43.838931 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.838905 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:43.883331 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883300 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca\") pod \"534638ae-3b4a-4b36-8171-5030e668df3e\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " Oct 31 03:59:43.883494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883360 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca\") pod \"534638ae-3b4a-4b36-8171-5030e668df3e\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " Oct 31 03:59:43.883494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883393 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca\") pod \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " Oct 31 03:59:43.883494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883462 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2\" (UniqueName: \"kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2\") pod \"534638ae-3b4a-4b36-8171-5030e668df3e\" (UID: \"534638ae-3b4a-4b36-8171-5030e668df3e\") " Oct 31 03:59:43.883494 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883487 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq\" (UniqueName: \"kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq\") pod \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " Oct 31 03:59:43.883733 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883517 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca\") pod \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\" (UID: \"d36123b2-5c3f-4f3f-83da-01acc9a4317e\") " Oct 31 03:59:43.883795 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883675 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca" pod "534638ae-3b4a-4b36-8171-5030e668df3e" (UID: "534638ae-3b4a-4b36-8171-5030e668df3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:43.883886 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883860 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca" pod "d36123b2-5c3f-4f3f-83da-01acc9a4317e" (UID: "d36123b2-5c3f-4f3f-83da-01acc9a4317e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:43.884049 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.883925 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:43.885813 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.885786 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca" pod "d36123b2-5c3f-4f3f-83da-01acc9a4317e" (UID: "d36123b2-5c3f-4f3f-83da-01acc9a4317e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:43.885924 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.885811 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2" pod "534638ae-3b4a-4b36-8171-5030e668df3e" (UID: "534638ae-3b4a-4b36-8171-5030e668df3e"). InnerVolumeSpecName "kube-api-access-wkdc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:43.886218 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.886194 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca" pod "534638ae-3b4a-4b36-8171-5030e668df3e" (UID: "534638ae-3b4a-4b36-8171-5030e668df3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:43.886273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.886213 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq" pod "d36123b2-5c3f-4f3f-83da-01acc9a4317e" (UID: "d36123b2-5c3f-4f3f-83da-01acc9a4317e"). InnerVolumeSpecName "kube-api-access-4nrsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:43.985536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.985444 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534638ae-3b4a-4b36-8171-5030e668df3e-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:43.985536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.985476 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:43.985536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.985487 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdc2\" (UniqueName: \"kubernetes.io/projected/534638ae-3b4a-4b36-8171-5030e668df3e-kube-api-access-wkdc2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:43.985536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.985496 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrsq\" (UniqueName: \"kubernetes.io/projected/d36123b2-5c3f-4f3f-83da-01acc9a4317e-kube-api-access-4nrsq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:43.985536 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:43.985505 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36123b2-5c3f-4f3f-83da-01acc9a4317e-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.119291 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.119259 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:44.269134 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.269005 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:44.664132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.664082 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" Oct 31 03:59:44.664132 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.664122 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts" event={"ID":"d36123b2-5c3f-4f3f-83da-01acc9a4317e","Type":"ContainerDied","Data":"439d9eb3a738db2e08c11c5cb4a3d8acf129548907925a5f5c95b89dde8c25d6"} Oct 31 03:59:44.664478 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.664180 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="439d9eb3a738db2e08c11c5cb4a3d8acf129548907925a5f5c95b89dde8c25d6" Oct 31 03:59:44.665709 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.665681 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" Oct 31 03:59:44.665830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.665695 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw" event={"ID":"534638ae-3b4a-4b36-8171-5030e668df3e","Type":"ContainerDied","Data":"21c3a1ef31436431df9879053edc0a226ad0c52b6f74464ec4d449d9cc59d396"} Oct 31 03:59:44.665830 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:44.665807 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c3a1ef31436431df9879053edc0a226ad0c52b6f74464ec4d449d9cc59d396" Oct 31 03:59:46.548176 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.548144 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:46.867972 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.867930 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:59:46.930734 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.930698 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume\") pod \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " Oct 31 03:59:46.930950 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.930835 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw\" (UniqueName: \"kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw\") pod \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\" (UID: \"fd32d418-67ed-4d6b-bacf-8b27be0664e5\") " Oct 31 03:59:46.931111 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.931068 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume" pod "fd32d418-67ed-4d6b-bacf-8b27be0664e5" (UID: "fd32d418-67ed-4d6b-bacf-8b27be0664e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:46.931267 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.931251 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd32d418-67ed-4d6b-bacf-8b27be0664e5-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:46.933199 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:46.933178 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw" pod "fd32d418-67ed-4d6b-bacf-8b27be0664e5" (UID: "fd32d418-67ed-4d6b-bacf-8b27be0664e5"). InnerVolumeSpecName "kube-api-access-f4blw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:47.031936 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.031888 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4blw\" (UniqueName: \"kubernetes.io/projected/fd32d418-67ed-4d6b-bacf-8b27be0664e5-kube-api-access-f4blw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:47.683778 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.683741 4488 generic.go:356] "Generic (PLEG): container finished" podID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerID="fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7" exitCode=0 Oct 31 03:59:47.684352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.683796 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" Oct 31 03:59:47.684352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.683834 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" event={"ID":"fd32d418-67ed-4d6b-bacf-8b27be0664e5","Type":"ContainerDied","Data":"fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7"} Oct 31 03:59:47.684352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.683907 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2" event={"ID":"fd32d418-67ed-4d6b-bacf-8b27be0664e5","Type":"ContainerDied","Data":"8687ee7001105119619fadb017592fd5ac784e672ae5a5f5870632f3884a166e"} Oct 31 03:59:47.684352 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.683963 4488 scope.go:122] "RemoveContainer" containerID="fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7" Oct 31 03:59:47.694340 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.694312 4488 scope.go:122] "RemoveContainer" containerID="fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7" Oct 31 03:59:47.694691 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:47.694664 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7\": container with ID starting with fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7 not found: ID does not exist" containerID="fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7" Oct 31 03:59:47.694787 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.694700 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7"} err="failed to get container status \"fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7\": rpc error: code = NotFound desc = could not find container \"fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7\": container with ID starting with fd5c55c0f27f9e6773b6d33b68838c0ede1bedc5bbe46e1397a8cac1410d98a7 not found: ID does not exist" Oct 31 03:59:47.849712 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.849676 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2"] Oct 31 03:59:47.899170 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:47.899137 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-909365-698957c747-xtrf2"] Oct 31 03:59:48.315295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:48.315264 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" path="/var/lib/kubelet/pods/fd32d418-67ed-4d6b-bacf-8b27be0664e5/volumes" Oct 31 03:59:50.280273 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.280243 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:50.443481 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.443451 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:50.793332 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793300 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m"] Oct 31 03:59:50.793691 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793679 4488 cpu_manager.go:414] "containerMap: removing container" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerName="payload-pre-processing" Oct 31 03:59:50.793753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793693 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerName="payload-pre-processing" Oct 31 03:59:50.793753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793712 4488 cpu_manager.go:414] "containerMap: removing container" podUID="534638ae-3b4a-4b36-8171-5030e668df3e" containerName="revoke-keys" Oct 31 03:59:50.793753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793717 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534638ae-3b4a-4b36-8171-5030e668df3e" containerName="revoke-keys" Oct 31 03:59:50.793753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793725 4488 cpu_manager.go:414] "containerMap: removing container" podUID="d36123b2-5c3f-4f3f-83da-01acc9a4317e" containerName="revoke-keys" Oct 31 03:59:50.793753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793729 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36123b2-5c3f-4f3f-83da-01acc9a4317e" containerName="revoke-keys" Oct 31 03:59:50.793927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793804 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd32d418-67ed-4d6b-bacf-8b27be0664e5" containerName="payload-pre-processing" Oct 31 03:59:50.793927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793813 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="534638ae-3b4a-4b36-8171-5030e668df3e" containerName="revoke-keys" Oct 31 03:59:50.793927 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.793818 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36123b2-5c3f-4f3f-83da-01acc9a4317e" containerName="revoke-keys" Oct 31 03:59:50.798068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.798052 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.875908 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.875874 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.876124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.876004 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktpc\" (UniqueName: \"kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.876124 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.876057 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.901990 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.901955 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:50.902233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.901980 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 03:59:50.902233 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.901980 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:50.902450 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.902031 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:50.904769 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.904743 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m"] Oct 31 03:59:50.977599 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.977569 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktpc\" (UniqueName: \"kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.977792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.977619 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.977792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.977694 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.978342 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.978320 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:50.980420 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:50.980398 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:51.016732 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.016677 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:51.040613 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.040579 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktpc\" (UniqueName: \"kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc\") pod \"maas-api-revoke-sub-ba12909d-wvc9m\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:51.111819 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.111783 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:51.515392 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.515363 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m"] Oct 31 03:59:51.517488 ip-10-0-143-24 kubenswrapper[4488]: W1031 03:59:51.517459 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ba5504_66db_44a5_aca0_438a39e9b801.slice/crio-e65259ead82c981f89b4f568992861031f60db718c01657b74a21f6add6ab41f WatchSource:0}: Error finding container e65259ead82c981f89b4f568992861031f60db718c01657b74a21f6add6ab41f: Status 404 returned error can't find the container with id e65259ead82c981f89b4f568992861031f60db718c01657b74a21f6add6ab41f Oct 31 03:59:51.710428 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.710398 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" event={"ID":"78ba5504-66db-44a5-aca0-438a39e9b801","Type":"ContainerStarted","Data":"e65259ead82c981f89b4f568992861031f60db718c01657b74a21f6add6ab41f"} Oct 31 03:59:51.889114 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.884575 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" containerID="cri-o://2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80" gracePeriod=45 Oct 31 03:59:51.892506 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:51.892334 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" containerID="cri-o://d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a" gracePeriod=45 Oct 31 03:59:52.157375 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.157350 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:59:52.182565 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.182540 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:59:52.302237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302199 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home\") pod \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " Oct 31 03:59:52.302237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302237 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm\") pod \"c61b3003-fce7-4b05-bc06-71d024715ad7\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " Oct 31 03:59:52.302503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302279 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj\" (UniqueName: \"kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj\") pod \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " Oct 31 03:59:52.302503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302312 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir\") pod \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " Oct 31 03:59:52.302503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302344 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache\") pod \"c61b3003-fce7-4b05-bc06-71d024715ad7\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " Oct 31 03:59:52.302503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302421 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs\") pod \"c61b3003-fce7-4b05-bc06-71d024715ad7\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " Oct 31 03:59:52.302503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302464 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir\") pod \"c61b3003-fce7-4b05-bc06-71d024715ad7\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " Oct 31 03:59:52.302503 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302501 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm\") pod \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " Oct 31 03:59:52.302834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302535 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home" pod "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" (UID: "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.302834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302555 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs\") pod \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " Oct 31 03:59:52.302834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302587 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2\" (UniqueName: \"kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2\") pod \"c61b3003-fce7-4b05-bc06-71d024715ad7\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " Oct 31 03:59:52.302834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302644 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home\") pod \"c61b3003-fce7-4b05-bc06-71d024715ad7\" (UID: \"c61b3003-fce7-4b05-bc06-71d024715ad7\") " Oct 31 03:59:52.302834 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.302756 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache\") pod \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\" (UID: \"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab\") " Oct 31 03:59:52.303457 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.303434 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.303697 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.303675 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache" pod "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" (UID: "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.304887 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.304705 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir" pod "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" (UID: "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.305280 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.305072 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache" pod "c61b3003-fce7-4b05-bc06-71d024715ad7" (UID: "c61b3003-fce7-4b05-bc06-71d024715ad7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.305468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.305412 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home" pod "c61b3003-fce7-4b05-bc06-71d024715ad7" (UID: "c61b3003-fce7-4b05-bc06-71d024715ad7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.305559 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.305479 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs" pod "c61b3003-fce7-4b05-bc06-71d024715ad7" (UID: "c61b3003-fce7-4b05-bc06-71d024715ad7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:52.305929 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.305906 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir" pod "c61b3003-fce7-4b05-bc06-71d024715ad7" (UID: "c61b3003-fce7-4b05-bc06-71d024715ad7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.306068 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.305953 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj" pod "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" (UID: "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab"). InnerVolumeSpecName "kube-api-access-tchjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:52.307043 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.307011 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs" pod "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" (UID: "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:52.307168 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.307124 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm" pod "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" (UID: "70af8e7a-c2a3-4fe5-a812-cf3ad41562ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.307230 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.307194 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm" pod "c61b3003-fce7-4b05-bc06-71d024715ad7" (UID: "c61b3003-fce7-4b05-bc06-71d024715ad7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:52.307876 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.307850 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2" pod "c61b3003-fce7-4b05-bc06-71d024715ad7" (UID: "c61b3003-fce7-4b05-bc06-71d024715ad7"). InnerVolumeSpecName "kube-api-access-qphx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412817 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b3003-fce7-4b05-bc06-71d024715ad7-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412853 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412867 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412879 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412892 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphx2\" (UniqueName: \"kubernetes.io/projected/c61b3003-fce7-4b05-bc06-71d024715ad7-kube-api-access-qphx2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412908 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412920 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412933 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412946 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchjj\" (UniqueName: \"kubernetes.io/projected/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-kube-api-access-tchjj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412960 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.413005 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.412972 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b3003-fce7-4b05-bc06-71d024715ad7-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.585939 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.585902 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:52.718295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.718199 4488 generic.go:356] "Generic (PLEG): container finished" podID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerID="d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a" exitCode=0 Oct 31 03:59:52.718295 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.718247 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" Oct 31 03:59:52.718554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.718289 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" event={"ID":"c61b3003-fce7-4b05-bc06-71d024715ad7","Type":"ContainerDied","Data":"d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a"} Oct 31 03:59:52.718554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.718342 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z" event={"ID":"c61b3003-fce7-4b05-bc06-71d024715ad7","Type":"ContainerDied","Data":"476f3bef09e75952d58ebceca7ed5b8ab5febd0ca1a4616a92f4f0c222b8e94e"} Oct 31 03:59:52.718554 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.718364 4488 scope.go:122] "RemoveContainer" containerID="d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a" Oct 31 03:59:52.720306 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.720282 4488 generic.go:356] "Generic (PLEG): container finished" podID="78ba5504-66db-44a5-aca0-438a39e9b801" containerID="ee4ba3c118168399423edb544dfe7f6a105bb5ad11139a8d1dc012faeb244fd3" exitCode=0 Oct 31 03:59:52.720468 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.720384 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" event={"ID":"78ba5504-66db-44a5-aca0-438a39e9b801","Type":"ContainerDied","Data":"ee4ba3c118168399423edb544dfe7f6a105bb5ad11139a8d1dc012faeb244fd3"} Oct 31 03:59:52.722008 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.721991 4488 generic.go:356] "Generic (PLEG): container finished" podID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerID="2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80" exitCode=0 Oct 31 03:59:52.722110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.722018 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" event={"ID":"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab","Type":"ContainerDied","Data":"2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80"} Oct 31 03:59:52.722110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.722031 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" Oct 31 03:59:52.722110 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.722036 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p" event={"ID":"70af8e7a-c2a3-4fe5-a812-cf3ad41562ab","Type":"ContainerDied","Data":"c946c5c5982bcf3fde795c4c999b7540bc2558efb8dfa891a56344c88ba613ca"} Oct 31 03:59:52.728403 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.728383 4488 scope.go:122] "RemoveContainer" containerID="d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a" Oct 31 03:59:52.728689 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:52.728669 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a\": container with ID starting with d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a not found: ID does not exist" containerID="d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a" Oct 31 03:59:52.728730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.728700 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a"} err="failed to get container status \"d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a\": rpc error: code = NotFound desc = could not find container \"d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a\": container with ID starting with d270d50ff8904bf5d7784d3e20ea4f446c8f836af4464d17bbc614b7fcd3aa2a not found: ID does not exist" Oct 31 03:59:52.728730 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.728718 4488 scope.go:122] "RemoveContainer" containerID="2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80" Oct 31 03:59:52.736903 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.736884 4488 scope.go:122] "RemoveContainer" containerID="2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80" Oct 31 03:59:52.737190 ip-10-0-143-24 kubenswrapper[4488]: E1031 03:59:52.737170 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80\": container with ID starting with 2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80 not found: ID does not exist" containerID="2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80" Oct 31 03:59:52.737257 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.737195 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80"} err="failed to get container status \"2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80\": rpc error: code = NotFound desc = could not find container \"2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80\": container with ID starting with 2276aa5fc41740e1f0e272b8c9b4f7065700ca4821630a92602aa8d8f328cd80 not found: ID does not exist" Oct 31 03:59:52.741673 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.741644 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:52.894002 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:52.893962 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:53.000753 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.000723 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z"] Oct 31 03:59:53.004758 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.004727 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-w2-dec4a0-kserve-84dcb7f44b-2x78z"] Oct 31 03:59:53.045530 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.045489 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:53.077237 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.077202 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt"] Oct 31 03:59:53.082885 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.082861 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b98893-whnbt"] Oct 31 03:59:53.114162 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.114123 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p"] Oct 31 03:59:53.118808 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.118782 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-dec4a0/distinct-2-w2-dec4a0-kserve-b7b57d66f-vpr6p"] Oct 31 03:59:53.499370 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.499336 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:53.871889 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:53.871867 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:54.030478 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.030375 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc\" (UniqueName: \"kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc\") pod \"78ba5504-66db-44a5-aca0-438a39e9b801\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " Oct 31 03:59:54.030665 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.030646 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca\") pod \"78ba5504-66db-44a5-aca0-438a39e9b801\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " Oct 31 03:59:54.030717 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.030700 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca\") pod \"78ba5504-66db-44a5-aca0-438a39e9b801\" (UID: \"78ba5504-66db-44a5-aca0-438a39e9b801\") " Oct 31 03:59:54.030970 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.030950 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca" pod "78ba5504-66db-44a5-aca0-438a39e9b801" (UID: "78ba5504-66db-44a5-aca0-438a39e9b801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:54.031195 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.031172 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:54.032791 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.032759 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc" pod "78ba5504-66db-44a5-aca0-438a39e9b801" (UID: "78ba5504-66db-44a5-aca0-438a39e9b801"). InnerVolumeSpecName "kube-api-access-nktpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:54.033079 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.033033 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca" pod "78ba5504-66db-44a5-aca0-438a39e9b801" (UID: "78ba5504-66db-44a5-aca0-438a39e9b801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:54.132105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.132036 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ba5504-66db-44a5-aca0-438a39e9b801-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:54.132105 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.132080 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktpc\" (UniqueName: \"kubernetes.io/projected/78ba5504-66db-44a5-aca0-438a39e9b801-kube-api-access-nktpc\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 03:59:54.316081 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.315998 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" path="/var/lib/kubelet/pods/70af8e7a-c2a3-4fe5-a812-cf3ad41562ab/volumes" Oct 31 03:59:54.316443 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.316428 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" path="/var/lib/kubelet/pods/c61b3003-fce7-4b05-bc06-71d024715ad7/volumes" Oct 31 03:59:54.316792 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.316780 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9026167-3bd6-4a76-83ce-ae2347fb41ec" path="/var/lib/kubelet/pods/f9026167-3bd6-4a76-83ce-ae2347fb41ec/volumes" Oct 31 03:59:54.736748 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.736718 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" Oct 31 03:59:54.736917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.736741 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m" event={"ID":"78ba5504-66db-44a5-aca0-438a39e9b801","Type":"ContainerDied","Data":"e65259ead82c981f89b4f568992861031f60db718c01657b74a21f6add6ab41f"} Oct 31 03:59:54.736917 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:54.736797 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e65259ead82c981f89b4f568992861031f60db718c01657b74a21f6add6ab41f" Oct 31 03:59:56.576323 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:56.576291 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:57.205676 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:57.205641 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:57.357856 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:57.357824 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:57.649610 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:57.649576 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 03:59:58.034224 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:58.034119 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:59.123163 ip-10-0-143-24 kubenswrapper[4488]: I1031 03:59:59.123128 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:00.167331 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167296 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt"] Oct 31 04:00:00.167886 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167866 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" Oct 31 04:00:00.168032 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167888 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" Oct 31 04:00:00.168032 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167925 4488 cpu_manager.go:414] "containerMap: removing container" podUID="78ba5504-66db-44a5-aca0-438a39e9b801" containerName="revoke-keys" Oct 31 04:00:00.168032 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167933 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ba5504-66db-44a5-aca0-438a39e9b801" containerName="revoke-keys" Oct 31 04:00:00.168032 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167965 4488 cpu_manager.go:414] "containerMap: removing container" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" Oct 31 04:00:00.168032 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.167972 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" Oct 31 04:00:00.168311 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.168127 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61b3003-fce7-4b05-bc06-71d024715ad7" containerName="main" Oct 31 04:00:00.168311 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.168145 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ba5504-66db-44a5-aca0-438a39e9b801" containerName="revoke-keys" Oct 31 04:00:00.168311 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.168155 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="70af8e7a-c2a3-4fe5-a812-cf3ad41562ab" containerName="main" Oct 31 04:00:00.171503 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.171481 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr"] Oct 31 04:00:00.171646 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.171626 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.174907 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.174888 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp"] Oct 31 04:00:00.175057 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.174991 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.178321 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.178300 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.224320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.224291 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:00:00.224539 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.224296 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:00:00.224539 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.224477 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:00:00.278494 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.278450 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt"] Oct 31 04:00:00.281523 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.281494 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr"] Oct 31 04:00:00.285175 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.285144 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp"] Oct 31 04:00:00.314410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.314380 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xgx\" (UniqueName: \"kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.314572 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.314545 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqvm\" (UniqueName: \"kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.314631 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.314589 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.314666 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.314631 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.314697 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.314666 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlkj\" (UniqueName: \"kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.314740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.314699 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.341430 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.341394 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:00.415914 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.415868 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.416192 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.415926 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlkj\" (UniqueName: \"kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.416192 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.415976 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.416192 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.416116 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xgx\" (UniqueName: \"kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.416378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.416279 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqvm\" (UniqueName: \"kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.416378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.416330 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.416996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.416951 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.417451 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.417357 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.417451 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.417359 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.480483 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.480449 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqvm\" (UniqueName: \"kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.480687 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.480443 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlkj\" (UniqueName: \"kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.480687 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.480595 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xgx\" (UniqueName: \"kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx\") pod \"maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.487337 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.487298 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:00.497649 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.497620 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:00.498797 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.498766 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:00.505056 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.505028 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:00.664158 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.664120 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67"] Oct 31 04:00:00.664381 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.664353 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" containerID="cri-o://8abfdd3a6a9e73a19464b505a4c794fb6ad1f93664d7d4df1886873a26f1d271" gracePeriod=30 Oct 31 04:00:00.705473 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.705389 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:00.925435 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.925398 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt"] Oct 31 04:00:00.926602 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:00:00.926574 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae41f81_08bd_4ae3_9a8d_1e45d8361274.slice/crio-40d8985164e9aaf83f2ffe649fd98c9140cfb6f8affa9723a5225f5755ce7ad3 WatchSource:0}: Error finding container 40d8985164e9aaf83f2ffe649fd98c9140cfb6f8affa9723a5225f5755ce7ad3: Status 404 returned error can't find the container with id 40d8985164e9aaf83f2ffe649fd98c9140cfb6f8affa9723a5225f5755ce7ad3 Oct 31 04:00:00.930251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.930227 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp"] Oct 31 04:00:00.931663 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:00:00.931640 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ac5b98_6b4d_468c_b24f_0904582fd776.slice/crio-e69ed5cec4b70c33eb2fe1acd275c14d8be07bc6c80644cafa022e588cca18ac WatchSource:0}: Error finding container e69ed5cec4b70c33eb2fe1acd275c14d8be07bc6c80644cafa022e588cca18ac: Status 404 returned error can't find the container with id e69ed5cec4b70c33eb2fe1acd275c14d8be07bc6c80644cafa022e588cca18ac Oct 31 04:00:00.975318 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:00.975287 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr"] Oct 31 04:00:00.976430 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:00:00.976398 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dc1ed1_fa06_43c6_9b6f_802f0101b2dc.slice/crio-e718d3f338692fc873f89545a186f4e492194c1844a21adaa9b74e6c1eef6fcc WatchSource:0}: Error finding container e718d3f338692fc873f89545a186f4e492194c1844a21adaa9b74e6c1eef6fcc: Status 404 returned error can't find the container with id e718d3f338692fc873f89545a186f4e492194c1844a21adaa9b74e6c1eef6fcc Oct 31 04:00:01.779929 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.779884 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerStarted","Data":"c6f0a54ecf3b42a06d9f4a3f284ee2381476d02285d696e633f877d43221dabd"} Oct 31 04:00:01.780490 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.779982 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerStarted","Data":"e718d3f338692fc873f89545a186f4e492194c1844a21adaa9b74e6c1eef6fcc"} Oct 31 04:00:01.783188 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.783147 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" event={"ID":"26ac5b98-6b4d-468c-b24f-0904582fd776","Type":"ContainerStarted","Data":"5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b"} Oct 31 04:00:01.783340 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.783206 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" event={"ID":"26ac5b98-6b4d-468c-b24f-0904582fd776","Type":"ContainerStarted","Data":"e69ed5cec4b70c33eb2fe1acd275c14d8be07bc6c80644cafa022e588cca18ac"} Oct 31 04:00:01.784788 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.784757 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" event={"ID":"8ae41f81-08bd-4ae3-9a8d-1e45d8361274","Type":"ContainerStarted","Data":"46169724c1d2d3aed821daee56aae9af66f5a9a278750f9a40bd5e047ece4ff3"} Oct 31 04:00:01.784920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.784797 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" event={"ID":"8ae41f81-08bd-4ae3-9a8d-1e45d8361274","Type":"ContainerStarted","Data":"40d8985164e9aaf83f2ffe649fd98c9140cfb6f8affa9723a5225f5755ce7ad3"} Oct 31 04:00:01.898828 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:01.898775 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" podStartSLOduration=1.898759204 podStartE2EDuration="1.898759204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:01.896593341 +0000 UTC m=+2429.693311602" watchObservedRunningTime="2026-10-31 04:00:01.898759204 +0000 UTC m=+2429.695477462" Oct 31 04:00:02.013315 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.013263 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" podStartSLOduration=2.013247133 podStartE2EDuration="2.013247133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:02.010860763 +0000 UTC m=+2429.807579021" watchObservedRunningTime="2026-10-31 04:00:02.013247133 +0000 UTC m=+2429.809965429" Oct 31 04:00:02.128410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.128354 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" podStartSLOduration=2.128336317 podStartE2EDuration="2.128336317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:02.126117512 +0000 UTC m=+2429.922835772" watchObservedRunningTime="2026-10-31 04:00:02.128336317 +0000 UTC m=+2429.925054635" Oct 31 04:00:02.275257 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.275225 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:02.341386 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.341352 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp"] Oct 31 04:00:02.792721 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.792682 4488 generic.go:356] "Generic (PLEG): container finished" podID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerID="c6f0a54ecf3b42a06d9f4a3f284ee2381476d02285d696e633f877d43221dabd" exitCode=60 Oct 31 04:00:02.793204 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.792778 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerDied","Data":"c6f0a54ecf3b42a06d9f4a3f284ee2381476d02285d696e633f877d43221dabd"} Oct 31 04:00:02.793204 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.793130 4488 scope.go:122] "RemoveContainer" containerID="c6f0a54ecf3b42a06d9f4a3f284ee2381476d02285d696e633f877d43221dabd" Oct 31 04:00:02.794574 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.794542 4488 generic.go:356] "Generic (PLEG): container finished" podID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerID="5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b" exitCode=60 Oct 31 04:00:02.794682 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.794574 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" event={"ID":"26ac5b98-6b4d-468c-b24f-0904582fd776","Type":"ContainerDied","Data":"5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b"} Oct 31 04:00:02.794812 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.794719 4488 scope.go:122] "RemoveContainer" containerID="5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b" Oct 31 04:00:02.797272 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.797246 4488 generic.go:356] "Generic (PLEG): container finished" podID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerID="46169724c1d2d3aed821daee56aae9af66f5a9a278750f9a40bd5e047ece4ff3" exitCode=60 Oct 31 04:00:02.797373 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.797271 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" event={"ID":"8ae41f81-08bd-4ae3-9a8d-1e45d8361274","Type":"ContainerDied","Data":"46169724c1d2d3aed821daee56aae9af66f5a9a278750f9a40bd5e047ece4ff3"} Oct 31 04:00:02.797640 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:02.797555 4488 scope.go:122] "RemoveContainer" containerID="46169724c1d2d3aed821daee56aae9af66f5a9a278750f9a40bd5e047ece4ff3" Oct 31 04:00:03.790729 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.790701 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:03.805394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.805353 4488 generic.go:356] "Generic (PLEG): container finished" podID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerID="b2841465825ee4c3cb829cfa2b9d386ef7d5c7f5f1a39ff959ef9b01dfdafc8b" exitCode=60 Oct 31 04:00:03.805870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.805446 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerDied","Data":"b2841465825ee4c3cb829cfa2b9d386ef7d5c7f5f1a39ff959ef9b01dfdafc8b"} Oct 31 04:00:03.805870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.805514 4488 scope.go:122] "RemoveContainer" containerID="c6f0a54ecf3b42a06d9f4a3f284ee2381476d02285d696e633f877d43221dabd" Oct 31 04:00:03.805870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.805768 4488 scope.go:122] "RemoveContainer" containerID="b2841465825ee4c3cb829cfa2b9d386ef7d5c7f5f1a39ff959ef9b01dfdafc8b" Oct 31 04:00:03.806155 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:03.806124 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr_odh-ai-gateway-infra(a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" Oct 31 04:00:03.807537 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.807511 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" event={"ID":"26ac5b98-6b4d-468c-b24f-0904582fd776","Type":"ContainerStarted","Data":"39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e"} Oct 31 04:00:03.807644 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.807558 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" containerID="cri-o://39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e" gracePeriod=30 Oct 31 04:00:03.809594 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.809573 4488 generic.go:356] "Generic (PLEG): container finished" podID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerID="78d9696bcec8c4233f896b2c0edac0e369f3424c6d93b0e50d44878626a4393f" exitCode=60 Oct 31 04:00:03.809701 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.809646 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" event={"ID":"8ae41f81-08bd-4ae3-9a8d-1e45d8361274","Type":"ContainerDied","Data":"78d9696bcec8c4233f896b2c0edac0e369f3424c6d93b0e50d44878626a4393f"} Oct 31 04:00:03.809936 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.809913 4488 scope.go:122] "RemoveContainer" containerID="78d9696bcec8c4233f896b2c0edac0e369f3424c6d93b0e50d44878626a4393f" Oct 31 04:00:03.810222 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:03.810203 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt_odh-ai-gateway-infra(8ae41f81-08bd-4ae3-9a8d-1e45d8361274)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" Oct 31 04:00:03.817613 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:03.817595 4488 scope.go:122] "RemoveContainer" containerID="46169724c1d2d3aed821daee56aae9af66f5a9a278750f9a40bd5e047ece4ff3" Oct 31 04:00:04.099981 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.099935 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:04.251643 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.251612 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:04.287841 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.287808 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph"] Oct 31 04:00:04.293477 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.293452 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909365-c6lph"] Oct 31 04:00:04.316330 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.316287 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80d7104-4cb2-4383-850a-13d975cd339f" path="/var/lib/kubelet/pods/a80d7104-4cb2-4383-850a-13d975cd339f/volumes" Oct 31 04:00:04.698845 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.698810 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:04.817435 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.817408 4488 scope.go:122] "RemoveContainer" containerID="78d9696bcec8c4233f896b2c0edac0e369f3424c6d93b0e50d44878626a4393f" Oct 31 04:00:04.817915 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:04.817725 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt_odh-ai-gateway-infra(8ae41f81-08bd-4ae3-9a8d-1e45d8361274)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" Oct 31 04:00:04.819211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:04.819192 4488 scope.go:122] "RemoveContainer" containerID="b2841465825ee4c3cb829cfa2b9d386ef7d5c7f5f1a39ff959ef9b01dfdafc8b" Oct 31 04:00:04.819443 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:04.819426 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr_odh-ai-gateway-infra(a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" Oct 31 04:00:05.827675 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.827644 4488 generic.go:356] "Generic (PLEG): container finished" podID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerID="8abfdd3a6a9e73a19464b505a4c794fb6ad1f93664d7d4df1886873a26f1d271" exitCode=0 Oct 31 04:00:05.827675 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.827674 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" event={"ID":"ab1eaf11-4e83-46a3-aec8-2f692d367be3","Type":"ContainerDied","Data":"8abfdd3a6a9e73a19464b505a4c794fb6ad1f93664d7d4df1886873a26f1d271"} Oct 31 04:00:05.828272 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.827749 4488 scope.go:122] "RemoveContainer" containerID="d5a9a81da646beb296b777f2587e39e9464ac603bbd84217081f798d6966dbd7" Oct 31 04:00:05.941613 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.941535 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 04:00:05.981414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981372 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981615 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981427 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981615 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981506 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981615 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981535 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981615 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981599 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981638 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981666 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981689 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:05.981847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981708 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.981847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981735 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket\") pod \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\" (UID: \"ab1eaf11-4e83-46a3-aec8-2f692d367be3\") " Oct 31 04:00:05.982161 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.981928 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:05.982378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.982318 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:05.982378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.982363 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:05.982526 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.982419 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:05.982689 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.982629 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:05.982867 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.982769 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:05.984341 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.984309 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:05.984341 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.984315 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:05.984488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.984405 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "kube-api-access-6g9rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:05.984906 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:05.984885 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy" pod "ab1eaf11-4e83-46a3-aec8-2f692d367be3" (UID: "ab1eaf11-4e83-46a3-aec8-2f692d367be3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:06.083792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083758 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.083792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083789 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.083792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083799 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.084081 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083807 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.084081 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083816 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.084081 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083829 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1eaf11-4e83-46a3-aec8-2f692d367be3-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.084081 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.083839 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9rw\" (UniqueName: \"kubernetes.io/projected/ab1eaf11-4e83-46a3-aec8-2f692d367be3-kube-api-access-6g9rw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:06.589991 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:06.589960 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:00:06.835048 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.835020 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" Oct 31 04:00:06.835507 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.835022 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67" event={"ID":"ab1eaf11-4e83-46a3-aec8-2f692d367be3","Type":"ContainerDied","Data":"f5eed4d0171df58feca7b59da31a95d0df947a64f7163b485dc4cd29edc53707"} Oct 31 04:00:06.835507 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:06.835159 4488 scope.go:122] "RemoveContainer" containerID="8abfdd3a6a9e73a19464b505a4c794fb6ad1f93664d7d4df1886873a26f1d271" Oct 31 04:00:07.000678 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:07.000637 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67"] Oct 31 04:00:07.050879 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:07.050844 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-jdd67"] Oct 31 04:00:08.315788 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:08.315757 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" path="/var/lib/kubelet/pods/ab1eaf11-4e83-46a3-aec8-2f692d367be3/volumes" Oct 31 04:00:09.215791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:09.215734 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:10.766961 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:10.766930 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:10.923392 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:10.923360 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:11.364967 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:11.364934 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:11.566420 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:11.566379 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:11.740439 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:11.740348 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:13.573399 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.573365 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:13.601801 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.601762 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt"] Oct 31 04:00:13.759402 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.759371 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:13.861374 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.861330 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca\") pod \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " Oct 31 04:00:13.861374 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.861379 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj\" (UniqueName: \"kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj\") pod \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\" (UID: \"8ae41f81-08bd-4ae3-9a8d-1e45d8361274\") " Oct 31 04:00:13.861717 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.861692 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca" pod "8ae41f81-08bd-4ae3-9a8d-1e45d8361274" (UID: "8ae41f81-08bd-4ae3-9a8d-1e45d8361274"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:13.861818 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.861802 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:13.863628 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.863605 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj" pod "8ae41f81-08bd-4ae3-9a8d-1e45d8361274" (UID: "8ae41f81-08bd-4ae3-9a8d-1e45d8361274"). InnerVolumeSpecName "kube-api-access-rmlkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:13.886569 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.886539 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" Oct 31 04:00:13.886791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.886575 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt" event={"ID":"8ae41f81-08bd-4ae3-9a8d-1e45d8361274","Type":"ContainerDied","Data":"40d8985164e9aaf83f2ffe649fd98c9140cfb6f8affa9723a5225f5755ce7ad3"} Oct 31 04:00:13.886791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.886654 4488 scope.go:122] "RemoveContainer" containerID="78d9696bcec8c4233f896b2c0edac0e369f3424c6d93b0e50d44878626a4393f" Oct 31 04:00:13.962906 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:13.962873 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlkj\" (UniqueName: \"kubernetes.io/projected/8ae41f81-08bd-4ae3-9a8d-1e45d8361274-kube-api-access-rmlkj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.063057 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:14.063020 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt"] Oct 31 04:00:14.070552 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:14.070521 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890320-spgzt"] Oct 31 04:00:14.316408 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:14.316327 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" path="/var/lib/kubelet/pods/8ae41f81-08bd-4ae3-9a8d-1e45d8361274/volumes" Oct 31 04:00:15.061076 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:15.061027 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:15.369777 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:15.369741 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:15.545068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:15.545030 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n"] Oct 31 04:00:15.575361 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:15.575322 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:15.594611 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:15.594580 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-dec4a0-s9g9n"] Oct 31 04:00:15.973518 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:15.973487 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:16.315907 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:16.315821 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee203307-1664-49e3-ae66-bad5d5c06167" path="/var/lib/kubelet/pods/ee203307-1664-49e3-ae66-bad5d5c06167/volumes" Oct 31 04:00:17.310317 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:17.310287 4488 scope.go:122] "RemoveContainer" containerID="b2841465825ee4c3cb829cfa2b9d386ef7d5c7f5f1a39ff959ef9b01dfdafc8b" Oct 31 04:00:17.911029 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:17.910947 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerStarted","Data":"a1d339d0d59df738a67777d7be72d1c28d8f1e0ad42f46dd2b70bd93cb76d3cb"} Oct 31 04:00:18.918441 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:18.918411 4488 generic.go:356] "Generic (PLEG): container finished" podID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerID="a1d339d0d59df738a67777d7be72d1c28d8f1e0ad42f46dd2b70bd93cb76d3cb" exitCode=60 Oct 31 04:00:18.918955 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:18.918485 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerDied","Data":"a1d339d0d59df738a67777d7be72d1c28d8f1e0ad42f46dd2b70bd93cb76d3cb"} Oct 31 04:00:18.918955 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:18.918549 4488 scope.go:122] "RemoveContainer" containerID="b2841465825ee4c3cb829cfa2b9d386ef7d5c7f5f1a39ff959ef9b01dfdafc8b" Oct 31 04:00:18.918955 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:18.918915 4488 scope.go:122] "RemoveContainer" containerID="a1d339d0d59df738a67777d7be72d1c28d8f1e0ad42f46dd2b70bd93cb76d3cb" Oct 31 04:00:18.919245 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:18.919222 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr_odh-ai-gateway-infra(a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" Oct 31 04:00:19.037384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:19.037345 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr"] Oct 31 04:00:20.081717 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.081695 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:20.127915 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.127877 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca\") pod \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " Oct 31 04:00:20.128160 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.127924 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm\" (UniqueName: \"kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm\") pod \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\" (UID: \"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc\") " Oct 31 04:00:20.128364 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.128339 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca" pod "a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" (UID: "a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:20.130368 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.130345 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm" pod "a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" (UID: "a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc"). InnerVolumeSpecName "kube-api-access-hxqvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:20.229802 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.229710 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:20.229802 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.229754 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqvm\" (UniqueName: \"kubernetes.io/projected/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc-kube-api-access-hxqvm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:20.507384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.507308 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:20.936794 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.936760 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" Oct 31 04:00:20.936995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.936792 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr" event={"ID":"a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc","Type":"ContainerDied","Data":"e718d3f338692fc873f89545a186f4e492194c1844a21adaa9b74e6c1eef6fcc"} Oct 31 04:00:20.936995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:20.936860 4488 scope.go:122] "RemoveContainer" containerID="a1d339d0d59df738a67777d7be72d1c28d8f1e0ad42f46dd2b70bd93cb76d3cb" Oct 31 04:00:21.108003 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:21.107961 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr"] Oct 31 04:00:21.112773 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:21.112742 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890320-xcprr"] Oct 31 04:00:22.316287 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:22.316253 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" path="/var/lib/kubelet/pods/a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc/volumes" Oct 31 04:00:23.079320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:23.079290 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:23.239059 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:23.239020 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:23.423364 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:23.423267 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:23.878211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:23.878179 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:24.032781 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:24.032746 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:24.277470 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:24.277393 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:24.951647 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:24.951605 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:25.193309 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:25.193274 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:25.413146 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:25.413074 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:26.970952 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.970915 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb"] Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971325 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971338 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971347 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971352 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971360 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971366 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971375 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971380 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971387 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.971421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971391 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.971771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971483 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" Oct 31 04:00:26.971771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971491 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.971771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971502 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.971771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971507 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerName="cleanup" Oct 31 04:00:26.971771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971513 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae41f81-08bd-4ae3-9a8d-1e45d8361274" containerName="cleanup" Oct 31 04:00:26.971771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.971519 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:26.976145 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:26.976122 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.033126 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.033075 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb"] Oct 31 04:00:27.080079 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.080048 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:00:27.121854 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.121812 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.122048 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.121940 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.122048 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.122031 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbnl\" (UniqueName: \"kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.223294 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.223202 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.223475 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.223331 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbnl\" (UniqueName: \"kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.223475 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.223391 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.224144 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.224012 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.225925 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.225899 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.284873 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.284831 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbnl\" (UniqueName: \"kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl\") pod \"maas-api-revoke-sub-51456da3-p7bsb\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.289635 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.289615 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:27.669660 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.669633 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb"] Oct 31 04:00:27.672484 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:00:27.672451 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088adc04_5b77_4390_8995_d645932f03ec.slice/crio-1984ba5099588b545a35f33dbc6a92252d152570acd6451a4f274f92feaa20b4 WatchSource:0}: Error finding container 1984ba5099588b545a35f33dbc6a92252d152570acd6451a4f274f92feaa20b4: Status 404 returned error can't find the container with id 1984ba5099588b545a35f33dbc6a92252d152570acd6451a4f274f92feaa20b4 Oct 31 04:00:27.984791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:27.984757 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" event={"ID":"088adc04-5b77-4390-8995-d645932f03ec","Type":"ContainerStarted","Data":"1984ba5099588b545a35f33dbc6a92252d152570acd6451a4f274f92feaa20b4"} Oct 31 04:00:28.992352 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:28.992255 4488 generic.go:356] "Generic (PLEG): container finished" podID="088adc04-5b77-4390-8995-d645932f03ec" containerID="bb48a82c602ed2fc3224570d359169e2445e0b5d91433bf35239b0eee66c46ab" exitCode=0 Oct 31 04:00:28.992750 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:28.992345 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" event={"ID":"088adc04-5b77-4390-8995-d645932f03ec","Type":"ContainerDied","Data":"bb48a82c602ed2fc3224570d359169e2445e0b5d91433bf35239b0eee66c46ab"} Oct 31 04:00:30.147052 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.147023 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:30.220797 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.220761 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:30.262138 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.259526 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca\") pod \"088adc04-5b77-4390-8995-d645932f03ec\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " Oct 31 04:00:30.262138 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.259703 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca\") pod \"088adc04-5b77-4390-8995-d645932f03ec\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " Oct 31 04:00:30.262138 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.259790 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl\" (UniqueName: \"kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl\") pod \"088adc04-5b77-4390-8995-d645932f03ec\" (UID: \"088adc04-5b77-4390-8995-d645932f03ec\") " Oct 31 04:00:30.262138 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.260383 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca" pod "088adc04-5b77-4390-8995-d645932f03ec" (UID: "088adc04-5b77-4390-8995-d645932f03ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:30.264022 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.263987 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl" pod "088adc04-5b77-4390-8995-d645932f03ec" (UID: "088adc04-5b77-4390-8995-d645932f03ec"). InnerVolumeSpecName "kube-api-access-vpbnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:30.264253 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.264072 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca" pod "088adc04-5b77-4390-8995-d645932f03ec" (UID: "088adc04-5b77-4390-8995-d645932f03ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:30.361369 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.361319 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088adc04-5b77-4390-8995-d645932f03ec-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:30.361369 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.361360 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088adc04-5b77-4390-8995-d645932f03ec-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:30.361369 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:30.361374 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpbnl\" (UniqueName: \"kubernetes.io/projected/088adc04-5b77-4390-8995-d645932f03ec-kube-api-access-vpbnl\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:31.005497 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:31.005467 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" Oct 31 04:00:31.005497 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:31.005476 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb" event={"ID":"088adc04-5b77-4390-8995-d645932f03ec","Type":"ContainerDied","Data":"1984ba5099588b545a35f33dbc6a92252d152570acd6451a4f274f92feaa20b4"} Oct 31 04:00:31.005737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:31.005526 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1984ba5099588b545a35f33dbc6a92252d152570acd6451a4f274f92feaa20b4" Oct 31 04:00:31.053604 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:31.053564 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:34.008180 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.008145 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:34.024680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.024642 4488 generic.go:356] "Generic (PLEG): container finished" podID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerID="39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e" exitCode=137 Oct 31 04:00:34.024874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.024693 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" Oct 31 04:00:34.024874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.024729 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" event={"ID":"26ac5b98-6b4d-468c-b24f-0904582fd776","Type":"ContainerDied","Data":"39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e"} Oct 31 04:00:34.024874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.024816 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp" event={"ID":"26ac5b98-6b4d-468c-b24f-0904582fd776","Type":"ContainerDied","Data":"e69ed5cec4b70c33eb2fe1acd275c14d8be07bc6c80644cafa022e588cca18ac"} Oct 31 04:00:34.024874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.024847 4488 scope.go:122] "RemoveContainer" containerID="39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e" Oct 31 04:00:34.034276 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.034257 4488 scope.go:122] "RemoveContainer" containerID="5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b" Oct 31 04:00:34.045042 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.045018 4488 scope.go:122] "RemoveContainer" containerID="39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e" Oct 31 04:00:34.045562 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:34.045543 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e\": container with ID starting with 39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e not found: ID does not exist" containerID="39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e" Oct 31 04:00:34.045671 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.045572 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e"} err="failed to get container status \"39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e\": rpc error: code = NotFound desc = could not find container \"39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e\": container with ID starting with 39dfb005b2c2d0bef4d475d5dc460c3f2dfff12ef06b35b5a861a6ede7aa1a3e not found: ID does not exist" Oct 31 04:00:34.045671 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.045591 4488 scope.go:122] "RemoveContainer" containerID="5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b" Oct 31 04:00:34.045893 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:34.045877 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b\": container with ID starting with 5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b not found: ID does not exist" containerID="5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b" Oct 31 04:00:34.045935 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.045898 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b"} err="failed to get container status \"5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b\": rpc error: code = NotFound desc = could not find container \"5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b\": container with ID starting with 5073373b5e65143c2a0e29c74d7a15c891e9bee545b31978b0a5c7eeb9a6375b not found: ID does not exist" Oct 31 04:00:34.099906 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.099868 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx\" (UniqueName: \"kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx\") pod \"26ac5b98-6b4d-468c-b24f-0904582fd776\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " Oct 31 04:00:34.100145 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.099954 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca\") pod \"26ac5b98-6b4d-468c-b24f-0904582fd776\" (UID: \"26ac5b98-6b4d-468c-b24f-0904582fd776\") " Oct 31 04:00:34.100378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.100351 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca" pod "26ac5b98-6b4d-468c-b24f-0904582fd776" (UID: "26ac5b98-6b4d-468c-b24f-0904582fd776"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:34.102204 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.102183 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx" pod "26ac5b98-6b4d-468c-b24f-0904582fd776" (UID: "26ac5b98-6b4d-468c-b24f-0904582fd776"). InnerVolumeSpecName "kube-api-access-z4xgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:34.201721 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.201685 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xgx\" (UniqueName: \"kubernetes.io/projected/26ac5b98-6b4d-468c-b24f-0904582fd776-kube-api-access-z4xgx\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:34.201721 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.201718 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac5b98-6b4d-468c-b24f-0904582fd776-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:34.439283 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.439188 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp"] Oct 31 04:00:34.443578 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:34.443546 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-909365-29890320-ll9pp"] Oct 31 04:00:35.100117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:35.100058 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:35.317198 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:35.317148 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:00:36.316729 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:36.316696 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" path="/var/lib/kubelet/pods/26ac5b98-6b4d-468c-b24f-0904582fd776/volumes" Oct 31 04:00:36.590342 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:00:36.590305 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:00:38.111350 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111317 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd"] Oct 31 04:00:38.111787 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111755 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" Oct 31 04:00:38.111787 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111769 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" Oct 31 04:00:38.111787 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111778 4488 cpu_manager.go:414] "containerMap: removing container" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:38.111787 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111783 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1dc1ed1-fa06-43c6-9b6f-802f0101b2dc" containerName="cleanup" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111799 4488 cpu_manager.go:414] "containerMap: removing container" podUID="088adc04-5b77-4390-8995-d645932f03ec" containerName="revoke-keys" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111805 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088adc04-5b77-4390-8995-d645932f03ec" containerName="revoke-keys" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111815 4488 cpu_manager.go:414] "containerMap: removing container" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111824 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111842 4488 cpu_manager.go:414] "containerMap: removing container" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111846 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111926 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1eaf11-4e83-46a3-aec8-2f692d367be3" containerName="istio-proxy" Oct 31 04:00:38.111949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111952 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" Oct 31 04:00:38.112228 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.111958 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="088adc04-5b77-4390-8995-d645932f03ec" containerName="revoke-keys" Oct 31 04:00:38.116727 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.116706 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.171209 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.171173 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:00:38.171209 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.171177 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:00:38.171488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.171192 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:00:38.171488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.171265 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:00:38.173824 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.173799 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd"] Oct 31 04:00:38.247716 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.247669 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkh9x\" (UniqueName: \"kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.247909 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.247802 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.247909 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.247847 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.348741 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.348693 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkh9x\" (UniqueName: \"kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.348940 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.348797 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.348940 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.348836 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.349709 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.349677 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.351486 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.351462 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.410419 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.410345 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkh9x\" (UniqueName: \"kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.429881 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.429847 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:38.807628 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:38.807600 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd"] Oct 31 04:00:38.809670 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:00:38.809635 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2623bb9f_2555_4447_9b16_bbd753a792cf.slice/crio-cd05ebf29fb618e006a994795874a3750957cba8e9f2d7778d0df55bd0392aaa WatchSource:0}: Error finding container cd05ebf29fb618e006a994795874a3750957cba8e9f2d7778d0df55bd0392aaa: Status 404 returned error can't find the container with id cd05ebf29fb618e006a994795874a3750957cba8e9f2d7778d0df55bd0392aaa Oct 31 04:00:39.054725 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:39.054691 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" event={"ID":"2623bb9f-2555-4447-9b16-bbd753a792cf","Type":"ContainerStarted","Data":"cd05ebf29fb618e006a994795874a3750957cba8e9f2d7778d0df55bd0392aaa"} Oct 31 04:00:40.062324 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:40.062069 4488 generic.go:356] "Generic (PLEG): container finished" podID="2623bb9f-2555-4447-9b16-bbd753a792cf" containerID="e27fa04500ef2104834c3f51a297621c6a23277fa189698c4ddfde2cd17d984d" exitCode=0 Oct 31 04:00:40.062324 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:40.062213 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" event={"ID":"2623bb9f-2555-4447-9b16-bbd753a792cf","Type":"ContainerDied","Data":"e27fa04500ef2104834c3f51a297621c6a23277fa189698c4ddfde2cd17d984d"} Oct 31 04:00:40.133882 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:40.133854 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:41.211994 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.211963 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:41.279839 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.279802 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x\" (UniqueName: \"kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x\") pod \"2623bb9f-2555-4447-9b16-bbd753a792cf\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " Oct 31 04:00:41.280053 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.279848 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca\") pod \"2623bb9f-2555-4447-9b16-bbd753a792cf\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " Oct 31 04:00:41.280053 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.280000 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca\") pod \"2623bb9f-2555-4447-9b16-bbd753a792cf\" (UID: \"2623bb9f-2555-4447-9b16-bbd753a792cf\") " Oct 31 04:00:41.280397 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.280376 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca" pod "2623bb9f-2555-4447-9b16-bbd753a792cf" (UID: "2623bb9f-2555-4447-9b16-bbd753a792cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:41.282138 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.282116 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca" pod "2623bb9f-2555-4447-9b16-bbd753a792cf" (UID: "2623bb9f-2555-4447-9b16-bbd753a792cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:41.282237 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.282206 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x" pod "2623bb9f-2555-4447-9b16-bbd753a792cf" (UID: "2623bb9f-2555-4447-9b16-bbd753a792cf"). InnerVolumeSpecName "kube-api-access-dkh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:41.381104 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.381050 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:41.381315 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.381121 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkh9x\" (UniqueName: \"kubernetes.io/projected/2623bb9f-2555-4447-9b16-bbd753a792cf-kube-api-access-dkh9x\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:41.381315 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:41.381140 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2623bb9f-2555-4447-9b16-bbd753a792cf-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:00:42.074468 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:42.074436 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" Oct 31 04:00:42.074468 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:42.074445 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd" event={"ID":"2623bb9f-2555-4447-9b16-bbd753a792cf","Type":"ContainerDied","Data":"cd05ebf29fb618e006a994795874a3750957cba8e9f2d7778d0df55bd0392aaa"} Oct 31 04:00:42.074750 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:42.074489 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd05ebf29fb618e006a994795874a3750957cba8e9f2d7778d0df55bd0392aaa" Oct 31 04:00:45.252197 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:45.251918 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:47.799447 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:47.799410 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:49.969752 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:49.969714 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:50.131347 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:50.131313 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:53.239518 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:53.239482 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:53.394428 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:53.394395 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:54.373871 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:54.373838 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:55.976748 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:55.976718 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:56.197495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:56.197461 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:56.350300 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:56.350271 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:56.477449 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:56.477393 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd"] Oct 31 04:00:56.527217 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:56.527181 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6de37a-n98kd"] Oct 31 04:00:56.940510 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:56.940473 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:58.316174 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:00:58.316141 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2623bb9f-2555-4447-9b16-bbd753a792cf" path="/var/lib/kubelet/pods/2623bb9f-2555-4447-9b16-bbd753a792cf/volumes" Oct 31 04:01:00.019822 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:00.019779 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:01:01.356829 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:01.356794 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:03.414141 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:03.414081 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:03.574118 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:03.574053 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:03.775412 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:03.775307 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:04.713370 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.713330 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25"] Oct 31 04:01:04.713784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.713756 4488 cpu_manager.go:414] "containerMap: removing container" podUID="2623bb9f-2555-4447-9b16-bbd753a792cf" containerName="revoke-keys" Oct 31 04:01:04.713784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.713767 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2623bb9f-2555-4447-9b16-bbd753a792cf" containerName="revoke-keys" Oct 31 04:01:04.713866 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.713824 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ac5b98-6b4d-468c-b24f-0904582fd776" containerName="cleanup" Oct 31 04:01:04.713866 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.713836 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="2623bb9f-2555-4447-9b16-bbd753a792cf" containerName="revoke-keys" Oct 31 04:01:04.716927 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.716911 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.769399 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.769365 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:01:04.819230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.819189 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:01:04.819230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.819188 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:01:04.819453 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.819189 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:01:04.821536 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.821516 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25"] Oct 31 04:01:04.881668 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.881630 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fbj\" (UniqueName: \"kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.881668 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.881674 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.881907 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.881697 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.983462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.983355 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fbj\" (UniqueName: \"kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.983462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.983407 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.983462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.983432 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.984200 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.984173 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:04.985946 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:04.985926 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:05.045182 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:05.045135 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fbj\" (UniqueName: \"kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:05.330728 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:05.330694 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:05.741694 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:05.741661 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25"] Oct 31 04:01:05.744138 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:01:05.744105 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09e4348_2e9f_44eb_b7db_57ca7e9f97d0.slice/crio-13417d165a138aaaf2fd291cc0733231504ccd675a5f62016c77066e5458bab1 WatchSource:0}: Error finding container 13417d165a138aaaf2fd291cc0733231504ccd675a5f62016c77066e5458bab1: Status 404 returned error can't find the container with id 13417d165a138aaaf2fd291cc0733231504ccd675a5f62016c77066e5458bab1 Oct 31 04:01:06.221408 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:06.221373 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" event={"ID":"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0","Type":"ContainerStarted","Data":"de2407a41b2dd8a274ef49547a99100f9f86f55a88a3c91fd24bb9f7a0b5d124"} Oct 31 04:01:06.221644 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:06.221430 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" event={"ID":"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0","Type":"ContainerStarted","Data":"13417d165a138aaaf2fd291cc0733231504ccd675a5f62016c77066e5458bab1"} Oct 31 04:01:06.339065 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:06.339007 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" podStartSLOduration=2.338990862 podStartE2EDuration="2.338990862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:06.336784628 +0000 UTC m=+2494.133502886" watchObservedRunningTime="2026-10-31 04:01:06.338990862 +0000 UTC m=+2494.135709119" Oct 31 04:01:06.550776 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:06.550739 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:06.602308 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:06.602250 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:01:07.228454 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:07.228417 4488 generic.go:356] "Generic (PLEG): container finished" podID="f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" containerID="de2407a41b2dd8a274ef49547a99100f9f86f55a88a3c91fd24bb9f7a0b5d124" exitCode=0 Oct 31 04:01:07.228858 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:07.228506 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" event={"ID":"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0","Type":"ContainerDied","Data":"de2407a41b2dd8a274ef49547a99100f9f86f55a88a3c91fd24bb9f7a0b5d124"} Oct 31 04:01:08.395429 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.395404 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:08.525965 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.525883 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj\" (UniqueName: \"kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj\") pod \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " Oct 31 04:01:08.526159 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.525973 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca\") pod \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " Oct 31 04:01:08.526159 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.526069 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca\") pod \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\" (UID: \"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0\") " Oct 31 04:01:08.526351 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.526327 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca" pod "f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" (UID: "f09e4348-2e9f-44eb-b7db-57ca7e9f97d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:01:08.528367 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.528338 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca" pod "f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" (UID: "f09e4348-2e9f-44eb-b7db-57ca7e9f97d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:08.528474 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.528379 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj" pod "f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" (UID: "f09e4348-2e9f-44eb-b7db-57ca7e9f97d0"). InnerVolumeSpecName "kube-api-access-82fbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:08.627234 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.627191 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:08.627234 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.627230 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82fbj\" (UniqueName: \"kubernetes.io/projected/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-kube-api-access-82fbj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:08.627234 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:08.627246 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:09.242686 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:09.242656 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" Oct 31 04:01:09.242854 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:09.242685 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25" event={"ID":"f09e4348-2e9f-44eb-b7db-57ca7e9f97d0","Type":"ContainerDied","Data":"13417d165a138aaaf2fd291cc0733231504ccd675a5f62016c77066e5458bab1"} Oct 31 04:01:09.242854 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:09.242737 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13417d165a138aaaf2fd291cc0733231504ccd675a5f62016c77066e5458bab1" Oct 31 04:01:11.804584 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:11.804547 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:16.667825 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:16.667788 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:16.837303 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:16.837269 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:21.517520 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:21.517480 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:25.789784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:25.789748 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:27.301275 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:27.301234 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:27.605224 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:27.605192 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:27.763418 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:27.763378 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:27.816717 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:27.816679 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25"] Oct 31 04:01:27.866283 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:27.866200 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8de398-vfb25"] Oct 31 04:01:28.259184 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:28.259073 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:28.315184 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:28.315154 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" path="/var/lib/kubelet/pods/f09e4348-2e9f-44eb-b7db-57ca7e9f97d0/volumes" Oct 31 04:01:31.373867 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:31.373826 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:01:31.406386 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:31.406354 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:32.706759 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:32.706728 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:34.311529 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:34.311493 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:34.466723 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:34.466695 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:34.646244 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:34.646203 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5"] Oct 31 04:01:34.646476 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:34.646436 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" containerID="cri-o://8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428" gracePeriod=30 Oct 31 04:01:34.665473 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:34.665437 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:35.147810 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:35.147779 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:36.590505 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:36.590466 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:01:37.908139 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:37.908107 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:39.905888 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:39.905864 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 04:01:40.062525 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062435 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062525 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062480 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062525 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062516 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062831 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062689 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062831 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062741 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062831 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062776 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062831 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062808 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.062831 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062810 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:40.063132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062837 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.063132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.062971 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:40.063132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063047 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket\") pod \"78b4571b-f72d-4116-afb5-f8c63fc23db7\" (UID: \"78b4571b-f72d-4116-afb5-f8c63fc23db7\") " Oct 31 04:01:40.063471 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063184 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:40.063471 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063376 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:01:40.063471 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063443 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.063630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063460 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.063630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063542 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b4571b-f72d-4116-afb5-f8c63fc23db7-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.063630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063553 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.063630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.063547 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:40.065009 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.064985 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:40.065377 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.065349 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:01:40.065493 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.065354 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "kube-api-access-599np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:40.065493 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.065383 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token" pod "78b4571b-f72d-4116-afb5-f8c63fc23db7" (UID: "78b4571b-f72d-4116-afb5-f8c63fc23db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:40.130678 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.130641 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts"] Oct 31 04:01:40.134187 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.134156 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7"] Oct 31 04:01:40.136907 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.136866 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt"] Oct 31 04:01:40.164422 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.164386 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.164422 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.164420 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.164698 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.164435 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.164698 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.164448 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.164698 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.164464 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599np\" (UniqueName: \"kubernetes.io/projected/78b4571b-f72d-4116-afb5-f8c63fc23db7-kube-api-access-599np\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.219196 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.219136 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:40.422424 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.422390 4488 generic.go:356] "Generic (PLEG): container finished" podID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerID="8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428" exitCode=0 Oct 31 04:01:40.422651 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.422441 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" Oct 31 04:01:40.422651 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.422464 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" event={"ID":"78b4571b-f72d-4116-afb5-f8c63fc23db7","Type":"ContainerDied","Data":"8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428"} Oct 31 04:01:40.422651 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.422535 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5" event={"ID":"78b4571b-f72d-4116-afb5-f8c63fc23db7","Type":"ContainerDied","Data":"53a1ea92b7ed1d0b34a86b96eba7ff8033899693c4220b343c722cffc8c45e08"} Oct 31 04:01:40.422651 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.422565 4488 scope.go:122] "RemoveContainer" containerID="8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428" Oct 31 04:01:40.432524 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.432505 4488 scope.go:122] "RemoveContainer" containerID="3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf" Oct 31 04:01:40.441838 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.441811 4488 scope.go:122] "RemoveContainer" containerID="8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428" Oct 31 04:01:40.442241 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:40.442217 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428\": container with ID starting with 8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428 not found: ID does not exist" containerID="8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428" Oct 31 04:01:40.442358 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.442254 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428"} err="failed to get container status \"8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428\": rpc error: code = NotFound desc = could not find container \"8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428\": container with ID starting with 8c615366c2df68ebc8c8f68b9a04db8f9bd72f74e17e15abda73725861724428 not found: ID does not exist" Oct 31 04:01:40.442358 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.442280 4488 scope.go:122] "RemoveContainer" containerID="3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf" Oct 31 04:01:40.442612 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:40.442590 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf\": container with ID starting with 3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf not found: ID does not exist" containerID="3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf" Oct 31 04:01:40.442713 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.442616 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf"} err="failed to get container status \"3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf\": rpc error: code = NotFound desc = could not find container \"3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf\": container with ID starting with 3d9523d8252e9ba1064a6f5ba446e9c8bd2413c8f4d5a73fce85207e02234caf not found: ID does not exist" Oct 31 04:01:40.606684 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.606654 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5"] Oct 31 04:01:40.611884 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:40.611854 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8de398-openshift-default-864b6b8c57-pz4n5"] Oct 31 04:01:42.315782 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:42.315745 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" path="/var/lib/kubelet/pods/78b4571b-f72d-4116-afb5-f8c63fc23db7/volumes" Oct 31 04:01:43.118066 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:43.118028 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:47.110383 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.110342 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd"] Oct 31 04:01:47.438192 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438075 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl"] Oct 31 04:01:47.438688 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438669 4488 cpu_manager.go:414] "containerMap: removing container" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" Oct 31 04:01:47.438740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438691 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" Oct 31 04:01:47.438740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438708 4488 cpu_manager.go:414] "containerMap: removing container" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" Oct 31 04:01:47.438740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438716 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" Oct 31 04:01:47.438843 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438766 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" containerName="revoke-keys" Oct 31 04:01:47.438843 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438773 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" containerName="revoke-keys" Oct 31 04:01:47.438908 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438867 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" Oct 31 04:01:47.438908 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.438888 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09e4348-2e9f-44eb-b7db-57ca7e9f97d0" containerName="revoke-keys" Oct 31 04:01:47.442387 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.442361 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.500263 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.500229 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:01:47.500533 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.500406 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:01:47.502919 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.502893 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl"] Oct 31 04:01:47.515393 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.515354 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:47.547638 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.547600 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:01:47.548736 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.547985 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:01:47.654227 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.654080 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.654456 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.654263 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.654456 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.654416 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szcxp\" (UniqueName: \"kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.687048 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.687014 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:47.755319 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.755215 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.755319 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.755276 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.755319 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.755310 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szcxp\" (UniqueName: \"kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.755942 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.755918 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.758164 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.758131 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:47.819405 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:47.819366 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szcxp\" (UniqueName: \"kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp\") pod \"maas-api-revoke-sub-89be5563-7mbrl\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:48.063385 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:48.063296 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:48.440538 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:48.440352 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl"] Oct 31 04:01:48.443743 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:01:48.443714 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa741add_ff69_481e_888d_914f3dfb72a5.slice/crio-550a97abcdc4267a2ec790c661102802a2ab9bbdadb4283a1e45dd077b21ec7e WatchSource:0}: Error finding container 550a97abcdc4267a2ec790c661102802a2ab9bbdadb4283a1e45dd077b21ec7e: Status 404 returned error can't find the container with id 550a97abcdc4267a2ec790c661102802a2ab9bbdadb4283a1e45dd077b21ec7e Oct 31 04:01:48.471325 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:48.471289 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" event={"ID":"aa741add-ff69-481e-888d-914f3dfb72a5","Type":"ContainerStarted","Data":"550a97abcdc4267a2ec790c661102802a2ab9bbdadb4283a1e45dd077b21ec7e"} Oct 31 04:01:49.479523 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:49.479490 4488 generic.go:356] "Generic (PLEG): container finished" podID="aa741add-ff69-481e-888d-914f3dfb72a5" containerID="458038ab37ab1a9f88695b6c0dc5bbb970dd461704ddb1fe7d54cce837fdae3c" exitCode=0 Oct 31 04:01:49.479959 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:49.479566 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" event={"ID":"aa741add-ff69-481e-888d-914f3dfb72a5","Type":"ContainerDied","Data":"458038ab37ab1a9f88695b6c0dc5bbb970dd461704ddb1fe7d54cce837fdae3c"} Oct 31 04:01:50.631458 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.631432 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:50.713952 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.713909 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp\" (UniqueName: \"kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp\") pod \"aa741add-ff69-481e-888d-914f3dfb72a5\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " Oct 31 04:01:50.714215 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.714058 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca\") pod \"aa741add-ff69-481e-888d-914f3dfb72a5\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " Oct 31 04:01:50.714215 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.714149 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca\") pod \"aa741add-ff69-481e-888d-914f3dfb72a5\" (UID: \"aa741add-ff69-481e-888d-914f3dfb72a5\") " Oct 31 04:01:50.714448 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.714415 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca" pod "aa741add-ff69-481e-888d-914f3dfb72a5" (UID: "aa741add-ff69-481e-888d-914f3dfb72a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:01:50.716233 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.716203 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp" pod "aa741add-ff69-481e-888d-914f3dfb72a5" (UID: "aa741add-ff69-481e-888d-914f3dfb72a5"). InnerVolumeSpecName "kube-api-access-szcxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:50.716350 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.716273 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca" pod "aa741add-ff69-481e-888d-914f3dfb72a5" (UID: "aa741add-ff69-481e-888d-914f3dfb72a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:50.815155 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.814970 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szcxp\" (UniqueName: \"kubernetes.io/projected/aa741add-ff69-481e-888d-914f3dfb72a5-kube-api-access-szcxp\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:50.815155 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.815030 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:50.815155 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:50.815040 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa741add-ff69-481e-888d-914f3dfb72a5-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:51.491819 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:51.491785 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" Oct 31 04:01:51.491819 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:51.491812 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl" event={"ID":"aa741add-ff69-481e-888d-914f3dfb72a5","Type":"ContainerDied","Data":"550a97abcdc4267a2ec790c661102802a2ab9bbdadb4283a1e45dd077b21ec7e"} Oct 31 04:01:51.492061 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:51.491860 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="550a97abcdc4267a2ec790c661102802a2ab9bbdadb4283a1e45dd077b21ec7e" Oct 31 04:01:52.111497 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:52.111419 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerName="payload-processing" containerID="cri-o://9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9" gracePeriod=25 Oct 31 04:01:55.156956 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.156909 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" containerID="cri-o://8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72" gracePeriod=45 Oct 31 04:01:55.159984 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.159944 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" containerID="cri-o://1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4" gracePeriod=45 Oct 31 04:01:55.161662 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.161624 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" containerID="cri-o://ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3" gracePeriod=45 Oct 31 04:01:55.436602 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.436572 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 04:01:55.456645 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.456620 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 04:01:55.469120 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.469075 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 04:01:55.572916 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.572881 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm\") pod \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " Oct 31 04:01:55.572916 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.572916 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache\") pod \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.572952 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache\") pod \"3d5d2cf4-f779-401f-986c-ed2e89685787\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573009 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs\") pod \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573055 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs\" (UniqueName: \"kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs\") pod \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573116 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs\") pod \"3d5d2cf4-f779-401f-986c-ed2e89685787\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573150 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6\" (UniqueName: \"kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6\") pod \"3d5d2cf4-f779-401f-986c-ed2e89685787\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573195 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7\" (UniqueName: \"kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7\") pod \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " Oct 31 04:01:55.573230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573202 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache" pod "04e96f43-d2e3-4015-bcda-9ae34c66b75b" (UID: "04e96f43-d2e3-4015-bcda-9ae34c66b75b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573236 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home\") pod \"3d5d2cf4-f779-401f-986c-ed2e89685787\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573316 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir\") pod \"3d5d2cf4-f779-401f-986c-ed2e89685787\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573343 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home\") pod \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573383 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache\") pod \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573411 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm\") pod \"3d5d2cf4-f779-401f-986c-ed2e89685787\" (UID: \"3d5d2cf4-f779-401f-986c-ed2e89685787\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573464 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir\") pod \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573502 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir\") pod \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573541 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm\") pod \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573581 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2\" (UniqueName: \"kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2\") pod \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\" (UID: \"9c55bc84-c08a-418e-b19a-b8b97fa4bab6\") " Oct 31 04:01:55.573632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573629 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home\") pod \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\" (UID: \"04e96f43-d2e3-4015-bcda-9ae34c66b75b\") " Oct 31 04:01:55.574186 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.573686 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache" pod "3d5d2cf4-f779-401f-986c-ed2e89685787" (UID: "3d5d2cf4-f779-401f-986c-ed2e89685787"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.574245 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.574206 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.574245 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.574227 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.574941 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.574907 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home" pod "9c55bc84-c08a-418e-b19a-b8b97fa4bab6" (UID: "9c55bc84-c08a-418e-b19a-b8b97fa4bab6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.575264 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.575170 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir" pod "3d5d2cf4-f779-401f-986c-ed2e89685787" (UID: "3d5d2cf4-f779-401f-986c-ed2e89685787"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.575464 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.575438 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir" pod "04e96f43-d2e3-4015-bcda-9ae34c66b75b" (UID: "04e96f43-d2e3-4015-bcda-9ae34c66b75b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.576720 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.575500 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home" pod "3d5d2cf4-f779-401f-986c-ed2e89685787" (UID: "3d5d2cf4-f779-401f-986c-ed2e89685787"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.576720 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.575676 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache" pod "9c55bc84-c08a-418e-b19a-b8b97fa4bab6" (UID: "9c55bc84-c08a-418e-b19a-b8b97fa4bab6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.576898 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.576789 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir" pod "9c55bc84-c08a-418e-b19a-b8b97fa4bab6" (UID: "9c55bc84-c08a-418e-b19a-b8b97fa4bab6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.576898 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.576857 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home" pod "04e96f43-d2e3-4015-bcda-9ae34c66b75b" (UID: "04e96f43-d2e3-4015-bcda-9ae34c66b75b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.577137 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.577109 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs" pod "9c55bc84-c08a-418e-b19a-b8b97fa4bab6" (UID: "9c55bc84-c08a-418e-b19a-b8b97fa4bab6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:55.577220 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.577135 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs" pod "04e96f43-d2e3-4015-bcda-9ae34c66b75b" (UID: "04e96f43-d2e3-4015-bcda-9ae34c66b75b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:55.580235 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.579848 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7" pod "04e96f43-d2e3-4015-bcda-9ae34c66b75b" (UID: "04e96f43-d2e3-4015-bcda-9ae34c66b75b"). InnerVolumeSpecName "kube-api-access-dtsm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:55.580347 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.580324 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs" pod "3d5d2cf4-f779-401f-986c-ed2e89685787" (UID: "3d5d2cf4-f779-401f-986c-ed2e89685787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:55.580776 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.580744 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6" pod "3d5d2cf4-f779-401f-986c-ed2e89685787" (UID: "3d5d2cf4-f779-401f-986c-ed2e89685787"). InnerVolumeSpecName "kube-api-access-64ph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:55.581080 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.581060 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm" pod "9c55bc84-c08a-418e-b19a-b8b97fa4bab6" (UID: "9c55bc84-c08a-418e-b19a-b8b97fa4bab6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.581623 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.581598 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm" pod "3d5d2cf4-f779-401f-986c-ed2e89685787" (UID: "3d5d2cf4-f779-401f-986c-ed2e89685787"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.581761 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.581724 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm" pod "04e96f43-d2e3-4015-bcda-9ae34c66b75b" (UID: "04e96f43-d2e3-4015-bcda-9ae34c66b75b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:55.585221 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.585195 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2" pod "9c55bc84-c08a-418e-b19a-b8b97fa4bab6" (UID: "9c55bc84-c08a-418e-b19a-b8b97fa4bab6"). InnerVolumeSpecName "kube-api-access-hzdc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:55.585966 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.585936 4488 generic.go:356] "Generic (PLEG): container finished" podID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerID="1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4" exitCode=0 Oct 31 04:01:55.586083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.586017 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" Oct 31 04:01:55.586083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.586045 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" event={"ID":"04e96f43-d2e3-4015-bcda-9ae34c66b75b","Type":"ContainerDied","Data":"1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4"} Oct 31 04:01:55.586262 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.586116 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7" event={"ID":"04e96f43-d2e3-4015-bcda-9ae34c66b75b","Type":"ContainerDied","Data":"3adf23777046d55cca529ef5e9de9977f6afbf924db3a2b9f10aeaa731111ba2"} Oct 31 04:01:55.586262 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.586116 4488 scope.go:122] "RemoveContainer" containerID="1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4" Oct 31 04:01:55.589285 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.588285 4488 generic.go:356] "Generic (PLEG): container finished" podID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerID="ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3" exitCode=0 Oct 31 04:01:55.589285 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.588326 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" Oct 31 04:01:55.589285 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.588472 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" event={"ID":"9c55bc84-c08a-418e-b19a-b8b97fa4bab6","Type":"ContainerDied","Data":"ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3"} Oct 31 04:01:55.589285 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.588501 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt" event={"ID":"9c55bc84-c08a-418e-b19a-b8b97fa4bab6","Type":"ContainerDied","Data":"60fbabb1eda5f2375173b124dc18979359fdc0f96919dbd70bbb362fc0ad6fd9"} Oct 31 04:01:55.590397 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.590373 4488 generic.go:356] "Generic (PLEG): container finished" podID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerID="8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72" exitCode=0 Oct 31 04:01:55.590512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.590408 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" event={"ID":"3d5d2cf4-f779-401f-986c-ed2e89685787","Type":"ContainerDied","Data":"8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72"} Oct 31 04:01:55.590512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.590419 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" Oct 31 04:01:55.590512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.590436 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts" event={"ID":"3d5d2cf4-f779-401f-986c-ed2e89685787","Type":"ContainerDied","Data":"d017d20e90c0138d419f086f685c76172e6b8fc6e72c7078e4615789bc3a5782"} Oct 31 04:01:55.599299 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.599281 4488 scope.go:122] "RemoveContainer" containerID="1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4" Oct 31 04:01:55.599925 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:55.599825 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4\": container with ID starting with 1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4 not found: ID does not exist" containerID="1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4" Oct 31 04:01:55.599925 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.599883 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4"} err="failed to get container status \"1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4\": rpc error: code = NotFound desc = could not find container \"1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4\": container with ID starting with 1e4201b4354f50891efee8c663e404a7ec958d0654fad986910dc79d1e83abf4 not found: ID does not exist" Oct 31 04:01:55.599925 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.599918 4488 scope.go:122] "RemoveContainer" containerID="ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3" Oct 31 04:01:55.610290 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.610269 4488 scope.go:122] "RemoveContainer" containerID="ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3" Oct 31 04:01:55.610638 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:55.610619 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3\": container with ID starting with ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3 not found: ID does not exist" containerID="ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3" Oct 31 04:01:55.610701 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.610645 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3"} err="failed to get container status \"ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3\": rpc error: code = NotFound desc = could not find container \"ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3\": container with ID starting with ad4de80b5a8c41137196d616b96b73801ab22cce5c07323689baac166a6a95e3 not found: ID does not exist" Oct 31 04:01:55.610701 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.610663 4488 scope.go:122] "RemoveContainer" containerID="8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72" Oct 31 04:01:55.620239 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.620211 4488 scope.go:122] "RemoveContainer" containerID="8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72" Oct 31 04:01:55.620633 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:01:55.620604 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72\": container with ID starting with 8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72 not found: ID does not exist" containerID="8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72" Oct 31 04:01:55.620711 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.620639 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72"} err="failed to get container status \"8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72\": rpc error: code = NotFound desc = could not find container \"8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72\": container with ID starting with 8d4d083cab7e8856897de95fa97894747fb47f595af4c4115daea4ab5aea7e72 not found: ID does not exist" Oct 31 04:01:55.675410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675330 4488 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-model-cache\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675358 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675383 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675393 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675401 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675410 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675409 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdc2\" (UniqueName: \"kubernetes.io/projected/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-kube-api-access-hzdc2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675418 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e96f43-d2e3-4015-bcda-9ae34c66b75b-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675426 4488 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-dshm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675443 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675451 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e96f43-d2e3-4015-bcda-9ae34c66b75b-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675459 4488 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5d2cf4-f779-401f-986c-ed2e89685787-tls-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675467 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ph6\" (UniqueName: \"kubernetes.io/projected/3d5d2cf4-f779-401f-986c-ed2e89685787-kube-api-access-64ph6\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675475 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtsm7\" (UniqueName: \"kubernetes.io/projected/04e96f43-d2e3-4015-bcda-9ae34c66b75b-kube-api-access-dtsm7\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675484 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675491 4488 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2cf4-f779-401f-986c-ed2e89685787-tmp-dir\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.675737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.675499 4488 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c55bc84-c08a-418e-b19a-b8b97fa4bab6-home\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:01:55.911555 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.911519 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts"] Oct 31 04:01:55.914422 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:55.914395 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-2-w5-14cb5a-kserve-bb59c6cd9-2v9ts"] Oct 31 04:01:56.022426 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.022332 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt"] Oct 31 04:01:56.025475 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.025442 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/premium-w5-14cb5a-kserve-698496f87-7dwkt"] Oct 31 04:01:56.134126 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.134060 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7"] Oct 31 04:01:56.140989 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.140955 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/distinct-w5-14cb5a-kserve-6459776446-q4wh7"] Oct 31 04:01:56.315469 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.315390 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" path="/var/lib/kubelet/pods/04e96f43-d2e3-4015-bcda-9ae34c66b75b/volumes" Oct 31 04:01:56.315847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.315772 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" path="/var/lib/kubelet/pods/3d5d2cf4-f779-401f-986c-ed2e89685787/volumes" Oct 31 04:01:56.316148 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:01:56.316133 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" path="/var/lib/kubelet/pods/9c55bc84-c08a-418e-b19a-b8b97fa4bab6/volumes" Oct 31 04:02:00.166690 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:00.166647 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:02:02.395993 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:02.395956 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:03.253591 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:03.253554 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:03.416050 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:03.416009 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:03.971680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:03.971647 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:04.008990 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:04.008949 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:02:04.256492 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:04.256393 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:04.911573 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:04.911534 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:06.589447 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:02:06.589409 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:02:10.165888 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:10.165846 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:02:11.388840 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:11.388808 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:11.542207 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:11.542174 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:11.761903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:11.761821 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:12.365930 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.365900 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 04:02:12.441605 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.441488 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume\") pod \"26c61412-24c1-4932-8e68-b3f4720c5f77\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " Oct 31 04:02:12.441605 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.441546 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq\" (UniqueName: \"kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq\") pod \"26c61412-24c1-4932-8e68-b3f4720c5f77\" (UID: \"26c61412-24c1-4932-8e68-b3f4720c5f77\") " Oct 31 04:02:12.442182 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.441981 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume" pod "26c61412-24c1-4932-8e68-b3f4720c5f77" (UID: "26c61412-24c1-4932-8e68-b3f4720c5f77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:12.443962 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.443934 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq" pod "26c61412-24c1-4932-8e68-b3f4720c5f77" (UID: "26c61412-24c1-4932-8e68-b3f4720c5f77"). InnerVolumeSpecName "kube-api-access-jwxgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:12.542311 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.542274 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c61412-24c1-4932-8e68-b3f4720c5f77-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:12.542311 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.542305 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxgq\" (UniqueName: \"kubernetes.io/projected/26c61412-24c1-4932-8e68-b3f4720c5f77-kube-api-access-jwxgq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:12.686735 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.686698 4488 generic.go:356] "Generic (PLEG): container finished" podID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerID="9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9" exitCode=0 Oct 31 04:02:12.686920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.686748 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" Oct 31 04:02:12.686920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.686788 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" event={"ID":"26c61412-24c1-4932-8e68-b3f4720c5f77","Type":"ContainerDied","Data":"9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9"} Oct 31 04:02:12.686920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.686848 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd" event={"ID":"26c61412-24c1-4932-8e68-b3f4720c5f77","Type":"ContainerDied","Data":"42fc96e297029d3e95ca60410b9f83d989bf72466ec4b3a98172b51dbdd6e3a2"} Oct 31 04:02:12.686920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.686875 4488 scope.go:122] "RemoveContainer" containerID="9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9" Oct 31 04:02:12.696428 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.696409 4488 scope.go:122] "RemoveContainer" containerID="9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9" Oct 31 04:02:12.696740 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:02:12.696719 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9\": container with ID starting with 9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9 not found: ID does not exist" containerID="9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9" Oct 31 04:02:12.696810 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.696748 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9"} err="failed to get container status \"9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9\": rpc error: code = NotFound desc = could not find container \"9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9\": container with ID starting with 9047c66f047e883ccc83ec9257fa5b20d02e40c9c5122b14a86c9fe07231a7c9 not found: ID does not exist" Oct 31 04:02:12.852340 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.852303 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd"] Oct 31 04:02:12.857517 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:12.857484 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-14cb5a-74d74bc8cc-75rpd"] Oct 31 04:02:14.314918 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:14.314837 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" path="/var/lib/kubelet/pods/26c61412-24c1-4932-8e68-b3f4720c5f77/volumes" Oct 31 04:02:16.709206 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:16.709078 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:16.877629 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:16.877594 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:18.195325 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:18.195285 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:19.366524 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:19.366487 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:02:24.303488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:24.303455 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:31.761041 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:31.761005 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:36.588246 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:02:36.588204 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:02:36.744179 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.744136 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g"] Oct 31 04:02:36.744383 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.744350 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerName="discovery" containerID="cri-o://41d21f04e8725cae44b29126c7349bff709dd09c1ee78dceb9bff523d43951d7" gracePeriod=30 Oct 31 04:02:36.859688 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.859662 4488 generic.go:356] "Generic (PLEG): container finished" podID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerID="41d21f04e8725cae44b29126c7349bff709dd09c1ee78dceb9bff523d43951d7" exitCode=0 Oct 31 04:02:36.860226 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.859732 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" event={"ID":"491878a4-0f7d-4970-8ebc-bffb94fcef66","Type":"ContainerDied","Data":"41d21f04e8725cae44b29126c7349bff709dd09c1ee78dceb9bff523d43951d7"} Oct 31 04:02:36.905514 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.905482 4488 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6c56g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:8080/ready\": dial tcp 10.133.0.136:8080: connect: connection refused" start-of-body= Oct 31 04:02:36.905690 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.905535 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.136:8080/ready\": dial tcp 10.133.0.136:8080: connect: connection refused" Oct 31 04:02:36.991518 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:36.991493 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 04:02:37.038748 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038715 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.038968 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038757 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.038968 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038796 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.038968 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038815 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.038968 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038865 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.038968 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038942 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.039256 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.038971 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token\") pod \"491878a4-0f7d-4970-8ebc-bffb94fcef66\" (UID: \"491878a4-0f7d-4970-8ebc-bffb94fcef66\") " Oct 31 04:02:37.039698 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.039660 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:37.042191 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042157 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "kube-api-access-vv28q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:37.042368 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042344 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:02:37.042599 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042519 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:37.042599 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042565 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:37.042754 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042643 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:37.042754 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042665 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert" pod "491878a4-0f7d-4970-8ebc-bffb94fcef66" (UID: "491878a4-0f7d-4970-8ebc-bffb94fcef66"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:37.042870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042840 4488 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-kubeconfig\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.042870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042863 4488 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-ca-configmap\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.044550 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042893 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv28q\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-kube-api-access-vv28q\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.044550 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042909 4488 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-cacerts\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.044550 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042925 4488 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/491878a4-0f7d-4970-8ebc-bffb94fcef66-local-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.044550 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042940 4488 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-csr-dns-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.044550 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.042961 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491878a4-0f7d-4970-8ebc-bffb94fcef66-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:37.866588 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.866556 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" event={"ID":"491878a4-0f7d-4970-8ebc-bffb94fcef66","Type":"ContainerDied","Data":"a4f08c8c9a02962c5bdf27da51364f82952434ec2e5495bba18fa75328a068b2"} Oct 31 04:02:37.866588 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.866579 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g" Oct 31 04:02:37.867132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:37.866611 4488 scope.go:122] "RemoveContainer" containerID="41d21f04e8725cae44b29126c7349bff709dd09c1ee78dceb9bff523d43951d7" Oct 31 04:02:38.087258 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:38.087218 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g"] Oct 31 04:02:38.137485 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:38.137394 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6c56g"] Oct 31 04:02:38.316038 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:38.315997 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" path="/var/lib/kubelet/pods/491878a4-0f7d-4970-8ebc-bffb94fcef66/volumes" Oct 31 04:02:44.474479 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474434 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb"] Oct 31 04:02:44.474903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474854 4488 cpu_manager.go:414] "containerMap: removing container" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" Oct 31 04:02:44.474903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474870 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" Oct 31 04:02:44.474903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474885 4488 cpu_manager.go:414] "containerMap: removing container" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerName="discovery" Oct 31 04:02:44.474903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474890 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerName="discovery" Oct 31 04:02:44.474903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474903 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474909 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474922 4488 cpu_manager.go:414] "containerMap: removing container" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474928 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474940 4488 cpu_manager.go:414] "containerMap: removing container" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerName="payload-processing" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474946 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerName="payload-processing" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474954 4488 cpu_manager.go:414] "containerMap: removing container" podUID="aa741add-ff69-481e-888d-914f3dfb72a5" containerName="revoke-keys" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.474958 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa741add-ff69-481e-888d-914f3dfb72a5" containerName="revoke-keys" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475019 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c55bc84-c08a-418e-b19a-b8b97fa4bab6" containerName="main" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475027 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c61412-24c1-4932-8e68-b3f4720c5f77" containerName="payload-processing" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475034 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa741add-ff69-481e-888d-914f3dfb72a5" containerName="revoke-keys" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475041 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b4571b-f72d-4116-afb5-f8c63fc23db7" containerName="istio-proxy" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475051 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e96f43-d2e3-4015-bcda-9ae34c66b75b" containerName="main" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475057 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="491878a4-0f7d-4970-8ebc-bffb94fcef66" containerName="discovery" Oct 31 04:02:44.475070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.475064 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5d2cf4-f779-401f-986c-ed2e89685787" containerName="main" Oct 31 04:02:44.479457 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.479438 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.536890 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.536854 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb"] Oct 31 04:02:44.583546 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.583508 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:02:44.583546 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.583514 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:02:44.583771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.583551 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:02:44.583771 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.583585 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:02:44.614321 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.614276 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxc8\" (UniqueName: \"kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.614505 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.614399 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.614505 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.614444 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.715634 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.715596 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxc8\" (UniqueName: \"kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.715830 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.715685 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.715830 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.715727 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.716502 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.716482 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.718257 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.718235 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.777396 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.777318 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxc8\" (UniqueName: \"kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8\") pod \"maas-api-revoke-sub-85237e45-ww8nb\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:44.792718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:44.792686 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:45.171614 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:45.171590 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb"] Oct 31 04:02:45.174229 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:02:45.174199 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb73361_563b_4996_aa37_669963e66b1d.slice/crio-1ac4f9bff44e0f10605d81ee03397ec5a4a2222d8e2c226f76b0f9c91d52f52e WatchSource:0}: Error finding container 1ac4f9bff44e0f10605d81ee03397ec5a4a2222d8e2c226f76b0f9c91d52f52e: Status 404 returned error can't find the container with id 1ac4f9bff44e0f10605d81ee03397ec5a4a2222d8e2c226f76b0f9c91d52f52e Oct 31 04:02:45.913491 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:45.913453 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" event={"ID":"3eb73361-563b-4996-aa37-669963e66b1d","Type":"ContainerStarted","Data":"270aea95d4892444d9f857a74c340e0df1e08f60175f3c35a1ef80d6a4eb7d1b"} Oct 31 04:02:45.913909 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:45.913508 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" event={"ID":"3eb73361-563b-4996-aa37-669963e66b1d","Type":"ContainerStarted","Data":"1ac4f9bff44e0f10605d81ee03397ec5a4a2222d8e2c226f76b0f9c91d52f52e"} Oct 31 04:02:46.029778 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:46.029694 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" podStartSLOduration=2.029676011 podStartE2EDuration="2.029676011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:46.029263141 +0000 UTC m=+2593.825981405" watchObservedRunningTime="2026-10-31 04:02:46.029676011 +0000 UTC m=+2593.826394268" Oct 31 04:02:46.920132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:46.920081 4488 generic.go:356] "Generic (PLEG): container finished" podID="3eb73361-563b-4996-aa37-669963e66b1d" containerID="270aea95d4892444d9f857a74c340e0df1e08f60175f3c35a1ef80d6a4eb7d1b" exitCode=0 Oct 31 04:02:46.920530 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:46.920152 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" event={"ID":"3eb73361-563b-4996-aa37-669963e66b1d","Type":"ContainerDied","Data":"270aea95d4892444d9f857a74c340e0df1e08f60175f3c35a1ef80d6a4eb7d1b"} Oct 31 04:02:48.061418 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.061389 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:48.146258 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.146217 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca\") pod \"3eb73361-563b-4996-aa37-669963e66b1d\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " Oct 31 04:02:48.146452 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.146326 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8\" (UniqueName: \"kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8\") pod \"3eb73361-563b-4996-aa37-669963e66b1d\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " Oct 31 04:02:48.146452 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.146386 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca\") pod \"3eb73361-563b-4996-aa37-669963e66b1d\" (UID: \"3eb73361-563b-4996-aa37-669963e66b1d\") " Oct 31 04:02:48.146752 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.146728 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca" pod "3eb73361-563b-4996-aa37-669963e66b1d" (UID: "3eb73361-563b-4996-aa37-669963e66b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:48.148505 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.148478 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca" pod "3eb73361-563b-4996-aa37-669963e66b1d" (UID: "3eb73361-563b-4996-aa37-669963e66b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:48.148669 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.148649 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8" pod "3eb73361-563b-4996-aa37-669963e66b1d" (UID: "3eb73361-563b-4996-aa37-669963e66b1d"). InnerVolumeSpecName "kube-api-access-9zxc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:48.247531 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.247435 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:48.247531 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.247469 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxc8\" (UniqueName: \"kubernetes.io/projected/3eb73361-563b-4996-aa37-669963e66b1d-kube-api-access-9zxc8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:48.247531 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.247483 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb73361-563b-4996-aa37-669963e66b1d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:02:48.941870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.941833 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" event={"ID":"3eb73361-563b-4996-aa37-669963e66b1d","Type":"ContainerDied","Data":"1ac4f9bff44e0f10605d81ee03397ec5a4a2222d8e2c226f76b0f9c91d52f52e"} Oct 31 04:02:48.942069 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.941876 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb" Oct 31 04:02:48.942069 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:48.941921 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac4f9bff44e0f10605d81ee03397ec5a4a2222d8e2c226f76b0f9c91d52f52e" Oct 31 04:02:56.852170 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:56.852127 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:59.509988 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:02:59.509953 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:06.587585 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:03:06.587542 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:03:07.181480 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:07.181436 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:07.346073 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:07.346034 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:17.226513 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:17.226479 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:32.105555 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:32.105516 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:36.588490 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:03:36.588454 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:03:47.361333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:47.361300 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:03:55.910780 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.910732 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j"] Oct 31 04:03:55.911343 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.911320 4488 cpu_manager.go:414] "containerMap: removing container" podUID="3eb73361-563b-4996-aa37-669963e66b1d" containerName="revoke-keys" Oct 31 04:03:55.911394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.911349 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb73361-563b-4996-aa37-669963e66b1d" containerName="revoke-keys" Oct 31 04:03:55.911541 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.911525 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb73361-563b-4996-aa37-669963e66b1d" containerName="revoke-keys" Oct 31 04:03:55.916293 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.916267 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:55.969119 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.969038 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:03:55.969119 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.969045 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:03:55.969923 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.969886 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:03:55.970076 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.969970 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:03:55.972125 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:55.972073 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j"] Oct 31 04:03:56.019002 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.018957 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.019224 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.019031 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqkt\" (UniqueName: \"kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.019224 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.019180 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.120068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.120022 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.120300 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.120078 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.120300 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.120154 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqkt\" (UniqueName: \"kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.120855 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.120833 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.122694 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.122666 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.181926 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.181838 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqkt\" (UniqueName: \"kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt\") pod \"maas-api-revoke-sub-c1e271a9-qz95j\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.230535 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.230493 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:56.609160 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:56.609129 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j"] Oct 31 04:03:56.611141 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:03:56.611076 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7974b239_bc52_407d_aec6_bcd8b1cc3471.slice/crio-90335b7ec7e202a659628b15326d498fba60b2549caf29193cf2cb661edb9214 WatchSource:0}: Error finding container 90335b7ec7e202a659628b15326d498fba60b2549caf29193cf2cb661edb9214: Status 404 returned error can't find the container with id 90335b7ec7e202a659628b15326d498fba60b2549caf29193cf2cb661edb9214 Oct 31 04:03:57.190739 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:57.190637 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:57.325779 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:57.325691 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" event={"ID":"7974b239-bc52-407d-aec6-bcd8b1cc3471","Type":"ContainerStarted","Data":"5035812e0df313495fcc71e525a0a6f9a285159fb32926a45ff2c7e4aa75b69c"} Oct 31 04:03:57.325996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:57.325849 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" event={"ID":"7974b239-bc52-407d-aec6-bcd8b1cc3471","Type":"ContainerStarted","Data":"90335b7ec7e202a659628b15326d498fba60b2549caf29193cf2cb661edb9214"} Oct 31 04:03:57.352061 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:57.352022 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:57.451261 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:57.451140 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" podStartSLOduration=2.451122063 podStartE2EDuration="2.451122063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:57.448704451 +0000 UTC m=+2665.245422709" watchObservedRunningTime="2026-10-31 04:03:57.451122063 +0000 UTC m=+2665.247840319" Oct 31 04:03:58.332964 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:58.332927 4488 generic.go:356] "Generic (PLEG): container finished" podID="7974b239-bc52-407d-aec6-bcd8b1cc3471" containerID="5035812e0df313495fcc71e525a0a6f9a285159fb32926a45ff2c7e4aa75b69c" exitCode=0 Oct 31 04:03:58.333460 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:58.333012 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" event={"ID":"7974b239-bc52-407d-aec6-bcd8b1cc3471","Type":"ContainerDied","Data":"5035812e0df313495fcc71e525a0a6f9a285159fb32926a45ff2c7e4aa75b69c"} Oct 31 04:03:59.484652 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.484621 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:03:59.557804 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.557765 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca\") pod \"7974b239-bc52-407d-aec6-bcd8b1cc3471\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " Oct 31 04:03:59.557804 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.557806 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca\") pod \"7974b239-bc52-407d-aec6-bcd8b1cc3471\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " Oct 31 04:03:59.558040 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.557879 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt\" (UniqueName: \"kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt\") pod \"7974b239-bc52-407d-aec6-bcd8b1cc3471\" (UID: \"7974b239-bc52-407d-aec6-bcd8b1cc3471\") " Oct 31 04:03:59.558318 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.558287 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca" pod "7974b239-bc52-407d-aec6-bcd8b1cc3471" (UID: "7974b239-bc52-407d-aec6-bcd8b1cc3471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:03:59.560174 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.560149 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca" pod "7974b239-bc52-407d-aec6-bcd8b1cc3471" (UID: "7974b239-bc52-407d-aec6-bcd8b1cc3471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:03:59.560307 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.560175 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt" pod "7974b239-bc52-407d-aec6-bcd8b1cc3471" (UID: "7974b239-bc52-407d-aec6-bcd8b1cc3471"). InnerVolumeSpecName "kube-api-access-csqkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:03:59.658637 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.658543 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqkt\" (UniqueName: \"kubernetes.io/projected/7974b239-bc52-407d-aec6-bcd8b1cc3471-kube-api-access-csqkt\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:03:59.658637 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.658576 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:03:59.658637 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:03:59.658588 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7974b239-bc52-407d-aec6-bcd8b1cc3471-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:00.345289 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:00.345262 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" Oct 31 04:04:00.345494 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:00.345284 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j" event={"ID":"7974b239-bc52-407d-aec6-bcd8b1cc3471","Type":"ContainerDied","Data":"90335b7ec7e202a659628b15326d498fba60b2549caf29193cf2cb661edb9214"} Oct 31 04:04:00.345494 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:00.345332 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90335b7ec7e202a659628b15326d498fba60b2549caf29193cf2cb661edb9214" Oct 31 04:04:05.054567 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:05.054528 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc"] Oct 31 04:04:05.104498 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:05.104464 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f59a5-cp9dc"] Oct 31 04:04:06.314758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:06.314720 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca09c97-0fd2-490b-8a4b-ef50aa274650" path="/var/lib/kubelet/pods/bca09c97-0fd2-490b-8a4b-ef50aa274650/volumes" Oct 31 04:04:06.589228 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:04:06.589192 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:04:08.812034 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:08.811998 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:04:19.664932 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:19.664883 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:04:27.315632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.315592 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v"] Oct 31 04:04:27.319792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.316644 4488 cpu_manager.go:414] "containerMap: removing container" podUID="7974b239-bc52-407d-aec6-bcd8b1cc3471" containerName="revoke-keys" Oct 31 04:04:27.319792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.316679 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7974b239-bc52-407d-aec6-bcd8b1cc3471" containerName="revoke-keys" Oct 31 04:04:27.319792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.316943 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="7974b239-bc52-407d-aec6-bcd8b1cc3471" containerName="revoke-keys" Oct 31 04:04:27.324763 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.324734 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.378585 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.378551 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v"] Oct 31 04:04:27.425248 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.425212 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:04:27.425465 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.425397 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:04:27.425465 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.425428 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:04:27.427318 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.427296 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:04:27.457844 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.457804 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.457963 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.457880 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.458001 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.457972 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xsh\" (UniqueName: \"kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.559578 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.559534 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.559806 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.559609 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.559806 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.559635 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xsh\" (UniqueName: \"kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.560386 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.560360 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.562110 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.562073 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.621373 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.621338 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xsh\" (UniqueName: \"kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh\") pod \"maas-api-revoke-sub-2e1f5b5f-kww9v\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:27.638209 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:27.638174 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:28.014681 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:28.014655 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v"] Oct 31 04:04:28.017528 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:04:28.017499 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40aa8c13_8d3a_45b5_8df1_c3d6b4546cac.slice/crio-4b19a082e24b4fe436fa9e26ede904430c401ff5ea36b44d6cc40a4b544f74eb WatchSource:0}: Error finding container 4b19a082e24b4fe436fa9e26ede904430c401ff5ea36b44d6cc40a4b544f74eb: Status 404 returned error can't find the container with id 4b19a082e24b4fe436fa9e26ede904430c401ff5ea36b44d6cc40a4b544f74eb Oct 31 04:04:28.514472 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:28.514439 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" event={"ID":"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac","Type":"ContainerStarted","Data":"09bbecc7508d9c705ebf012c518498f78406fe103290e463e9e2d3a1ee4678f5"} Oct 31 04:04:28.514894 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:28.514500 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" event={"ID":"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac","Type":"ContainerStarted","Data":"4b19a082e24b4fe436fa9e26ede904430c401ff5ea36b44d6cc40a4b544f74eb"} Oct 31 04:04:28.681475 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:28.681411 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" podStartSLOduration=1.681395925 podStartE2EDuration="1.681395925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:28.632373693 +0000 UTC m=+2696.429091951" watchObservedRunningTime="2026-10-31 04:04:28.681395925 +0000 UTC m=+2696.478114207" Oct 31 04:04:29.521668 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:29.521636 4488 generic.go:356] "Generic (PLEG): container finished" podID="40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" containerID="09bbecc7508d9c705ebf012c518498f78406fe103290e463e9e2d3a1ee4678f5" exitCode=0 Oct 31 04:04:29.522166 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:29.521730 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" event={"ID":"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac","Type":"ContainerDied","Data":"09bbecc7508d9c705ebf012c518498f78406fe103290e463e9e2d3a1ee4678f5"} Oct 31 04:04:30.664187 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.664158 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:30.793657 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.793572 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh\" (UniqueName: \"kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh\") pod \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " Oct 31 04:04:30.793841 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.793692 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca\") pod \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " Oct 31 04:04:30.793841 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.793746 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca\") pod \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\" (UID: \"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac\") " Oct 31 04:04:30.794071 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.794031 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca" pod "40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" (UID: "40aa8c13-8d3a-45b5-8df1-c3d6b4546cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:30.794176 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.794152 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:30.796039 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.796012 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca" pod "40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" (UID: "40aa8c13-8d3a-45b5-8df1-c3d6b4546cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:30.796207 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.796189 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh" pod "40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" (UID: "40aa8c13-8d3a-45b5-8df1-c3d6b4546cac"). InnerVolumeSpecName "kube-api-access-p9xsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:30.895607 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.895568 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9xsh\" (UniqueName: \"kubernetes.io/projected/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-kube-api-access-p9xsh\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:30.895607 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:30.895602 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:31.541149 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:31.541116 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" Oct 31 04:04:31.541383 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:31.541143 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v" event={"ID":"40aa8c13-8d3a-45b5-8df1-c3d6b4546cac","Type":"ContainerDied","Data":"4b19a082e24b4fe436fa9e26ede904430c401ff5ea36b44d6cc40a4b544f74eb"} Oct 31 04:04:31.541383 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:31.541227 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b19a082e24b4fe436fa9e26ede904430c401ff5ea36b44d6cc40a4b544f74eb" Oct 31 04:04:32.048281 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:32.048244 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m"] Oct 31 04:04:32.098576 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:32.098543 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74787b76-4rn8m"] Oct 31 04:04:32.315404 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:32.315328 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d291a63e-2d80-4613-bfaa-2ee7e254d19a" path="/var/lib/kubelet/pods/d291a63e-2d80-4613-bfaa-2ee7e254d19a/volumes" Oct 31 04:04:32.495967 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:32.495929 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:04:33.047348 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:33.047316 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v"] Oct 31 04:04:33.097189 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:33.097151 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8432d7-ntq7v"] Oct 31 04:04:34.315214 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:34.315179 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f317710a-ecc1-456f-adc7-106258083322" path="/var/lib/kubelet/pods/f317710a-ecc1-456f-adc7-106258083322/volumes" Oct 31 04:04:35.540006 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:35.539976 4488 scope.go:122] "RemoveContainer" containerID="4a70bc8d5726cce3db1a7cf3c1f715b84e399befde5164e8297560434e2cc531" Oct 31 04:04:36.590417 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:04:36.590380 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:04:39.796994 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.796954 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr"] Oct 31 04:04:39.797429 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.797339 4488 cpu_manager.go:414] "containerMap: removing container" podUID="40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" containerName="revoke-keys" Oct 31 04:04:39.797429 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.797351 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" containerName="revoke-keys" Oct 31 04:04:39.797429 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.797422 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" containerName="revoke-keys" Oct 31 04:04:39.801828 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.801808 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.880063 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.880026 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.880291 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.880109 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.880291 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.880207 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7lc9\" (UniqueName: \"kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.906360 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.906325 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:04:39.906360 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.906353 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:04:39.906648 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.906530 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:04:39.906648 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.906560 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:04:39.908782 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.908763 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr"] Oct 31 04:04:39.981726 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.981684 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.981929 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.981763 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.981929 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.981786 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7lc9\" (UniqueName: \"kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.982369 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.982351 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:39.984426 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:39.984401 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:40.043772 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:40.043733 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7lc9\" (UniqueName: \"kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9\") pod \"maas-api-revoke-sub-ec843018-jhvrr\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:40.115687 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:40.115659 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:40.499510 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:40.499485 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr"] Oct 31 04:04:40.502182 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:04:40.502141 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81dafaa_66ea_46e1_a00f_9ddba5fb6560.slice/crio-95832545ed38bf69a0823fc3da379ea7c590629ac4d6cb6f571ce4b963b94909 WatchSource:0}: Error finding container 95832545ed38bf69a0823fc3da379ea7c590629ac4d6cb6f571ce4b963b94909: Status 404 returned error can't find the container with id 95832545ed38bf69a0823fc3da379ea7c590629ac4d6cb6f571ce4b963b94909 Oct 31 04:04:40.591366 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:40.591333 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" event={"ID":"c81dafaa-66ea-46e1-a00f-9ddba5fb6560","Type":"ContainerStarted","Data":"95832545ed38bf69a0823fc3da379ea7c590629ac4d6cb6f571ce4b963b94909"} Oct 31 04:04:41.598241 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:41.598203 4488 generic.go:356] "Generic (PLEG): container finished" podID="c81dafaa-66ea-46e1-a00f-9ddba5fb6560" containerID="0021aa18729897d9734c3fcdfd6d63175f552935d9c82b2c2d1246c38d6a38dc" exitCode=0 Oct 31 04:04:41.598678 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:41.598288 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" event={"ID":"c81dafaa-66ea-46e1-a00f-9ddba5fb6560","Type":"ContainerDied","Data":"0021aa18729897d9734c3fcdfd6d63175f552935d9c82b2c2d1246c38d6a38dc"} Oct 31 04:04:42.745164 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.745137 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:42.809996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.809965 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9\" (UniqueName: \"kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9\") pod \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " Oct 31 04:04:42.810203 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.810128 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca\") pod \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " Oct 31 04:04:42.810203 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.810181 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca\") pod \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\" (UID: \"c81dafaa-66ea-46e1-a00f-9ddba5fb6560\") " Oct 31 04:04:42.810541 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.810512 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca" pod "c81dafaa-66ea-46e1-a00f-9ddba5fb6560" (UID: "c81dafaa-66ea-46e1-a00f-9ddba5fb6560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:04:42.812226 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.812200 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9" pod "c81dafaa-66ea-46e1-a00f-9ddba5fb6560" (UID: "c81dafaa-66ea-46e1-a00f-9ddba5fb6560"). InnerVolumeSpecName "kube-api-access-l7lc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:04:42.812323 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.812293 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca" pod "c81dafaa-66ea-46e1-a00f-9ddba5fb6560" (UID: "c81dafaa-66ea-46e1-a00f-9ddba5fb6560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:04:42.911806 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.911719 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.911806 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.911752 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:42.911806 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:42.911762 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7lc9\" (UniqueName: \"kubernetes.io/projected/c81dafaa-66ea-46e1-a00f-9ddba5fb6560-kube-api-access-l7lc9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:04:43.608814 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:43.608780 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" event={"ID":"c81dafaa-66ea-46e1-a00f-9ddba5fb6560","Type":"ContainerDied","Data":"95832545ed38bf69a0823fc3da379ea7c590629ac4d6cb6f571ce4b963b94909"} Oct 31 04:04:43.608814 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:43.608795 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr" Oct 31 04:04:43.609047 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:43.608828 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95832545ed38bf69a0823fc3da379ea7c590629ac4d6cb6f571ce4b963b94909" Oct 31 04:04:43.779217 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:43.779185 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:04:44.051168 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:44.051016 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw"] Oct 31 04:04:44.053399 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:44.053369 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts"] Oct 31 04:04:44.056694 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:44.056669 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fdbde72-h62kw"] Oct 31 04:04:44.101011 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:44.100974 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63184e1b-tgcts"] Oct 31 04:04:44.315274 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:44.315184 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534638ae-3b4a-4b36-8171-5030e668df3e" path="/var/lib/kubelet/pods/534638ae-3b4a-4b36-8171-5030e668df3e/volumes" Oct 31 04:04:44.315636 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:44.315621 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36123b2-5c3f-4f3f-83da-01acc9a4317e" path="/var/lib/kubelet/pods/d36123b2-5c3f-4f3f-83da-01acc9a4317e/volumes" Oct 31 04:04:46.991983 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:46.991944 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:04:47.153811 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:47.153773 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:04:54.049320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:54.049280 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m"] Oct 31 04:04:54.097885 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:54.097846 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba12909d-wvc9m"] Oct 31 04:04:54.315292 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:54.315212 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ba5504-66db-44a5-aca0-438a39e9b801" path="/var/lib/kubelet/pods/78ba5504-66db-44a5-aca0-438a39e9b801/volumes" Oct 31 04:04:57.204344 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:04:57.204306 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:05:06.588746 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:05:06.588707 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:05:08.532928 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:08.532892 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:05:08.697164 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:08.697131 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:05:14.934718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:14.934625 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb"] Oct 31 04:05:14.935340 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:14.935219 4488 cpu_manager.go:414] "containerMap: removing container" podUID="c81dafaa-66ea-46e1-a00f-9ddba5fb6560" containerName="revoke-keys" Oct 31 04:05:14.935340 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:14.935239 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81dafaa-66ea-46e1-a00f-9ddba5fb6560" containerName="revoke-keys" Oct 31 04:05:14.935340 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:14.935323 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81dafaa-66ea-46e1-a00f-9ddba5fb6560" containerName="revoke-keys" Oct 31 04:05:14.938547 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:14.938525 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.006356 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.006318 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.006549 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.006513 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhff\" (UniqueName: \"kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.006631 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.006550 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.041413 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.041373 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:05:15.041602 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.041462 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:05:15.041602 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.041492 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:05:15.041602 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.041537 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:05:15.043435 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.043411 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb"] Oct 31 04:05:15.107879 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.107838 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhff\" (UniqueName: \"kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.108126 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.107888 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.108126 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.107985 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.108731 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.108705 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.110542 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.110520 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.170667 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.170634 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhff\" (UniqueName: \"kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff\") pod \"maas-api-revoke-sub-046e1343-vxgkb\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.251910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.251826 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:15.631668 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.631637 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb"] Oct 31 04:05:15.634987 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:05:15.634961 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a2e2f3_0195_400d_af7a_4ed3af59d8a9.slice/crio-4a79750150a069d34e18c3083a8b9a270ea4a7ea2072788a31ec0f184ee8e836 WatchSource:0}: Error finding container 4a79750150a069d34e18c3083a8b9a270ea4a7ea2072788a31ec0f184ee8e836: Status 404 returned error can't find the container with id 4a79750150a069d34e18c3083a8b9a270ea4a7ea2072788a31ec0f184ee8e836 Oct 31 04:05:15.799947 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:15.799903 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" event={"ID":"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9","Type":"ContainerStarted","Data":"4a79750150a069d34e18c3083a8b9a270ea4a7ea2072788a31ec0f184ee8e836"} Oct 31 04:05:16.806570 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:16.806476 4488 generic.go:356] "Generic (PLEG): container finished" podID="b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" containerID="a9f6aa32800bf2fdab7ecb1fc43e6fc1c2717e8eb83f929a6cbf1f21c640037e" exitCode=0 Oct 31 04:05:16.806570 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:16.806557 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" event={"ID":"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9","Type":"ContainerDied","Data":"a9f6aa32800bf2fdab7ecb1fc43e6fc1c2717e8eb83f929a6cbf1f21c640037e"} Oct 31 04:05:17.950316 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:17.950286 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:18.036241 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.036209 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca\") pod \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " Oct 31 04:05:18.036414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.036259 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca\") pod \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " Oct 31 04:05:18.036414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.036284 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff\" (UniqueName: \"kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff\") pod \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\" (UID: \"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9\") " Oct 31 04:05:18.036631 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.036607 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca" pod "b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" (UID: "b8a2e2f3-0195-400d-af7a-4ed3af59d8a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:18.038579 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.038548 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca" pod "b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" (UID: "b8a2e2f3-0195-400d-af7a-4ed3af59d8a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:18.038579 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.038555 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff" pod "b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" (UID: "b8a2e2f3-0195-400d-af7a-4ed3af59d8a9"). InnerVolumeSpecName "kube-api-access-zdhff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:18.137106 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.137061 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:05:18.137299 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.137127 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:05:18.137299 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.137139 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdhff\" (UniqueName: \"kubernetes.io/projected/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9-kube-api-access-zdhff\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:05:18.818423 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.818392 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" Oct 31 04:05:18.818630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.818393 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb" event={"ID":"b8a2e2f3-0195-400d-af7a-4ed3af59d8a9","Type":"ContainerDied","Data":"4a79750150a069d34e18c3083a8b9a270ea4a7ea2072788a31ec0f184ee8e836"} Oct 31 04:05:18.818630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:18.818515 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a79750150a069d34e18c3083a8b9a270ea4a7ea2072788a31ec0f184ee8e836" Oct 31 04:05:25.801058 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.801019 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx"] Oct 31 04:05:25.801624 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.801571 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" containerName="revoke-keys" Oct 31 04:05:25.801624 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.801591 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" containerName="revoke-keys" Oct 31 04:05:25.801759 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.801729 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" containerName="revoke-keys" Oct 31 04:05:25.805063 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.805038 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:25.905671 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.905627 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrggq\" (UniqueName: \"kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:25.905920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.905687 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:25.905920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.905803 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:25.910394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.910362 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:05:25.910532 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.910411 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:05:25.910532 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.910418 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:05:25.910532 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.910436 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:05:25.912619 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:25.912594 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx"] Oct 31 04:05:26.006957 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.006912 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrggq\" (UniqueName: \"kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.006957 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.006964 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.007286 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.006990 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.007662 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.007644 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.009518 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.009498 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.068978 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.068892 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrggq\" (UniqueName: \"kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq\") pod \"maas-api-revoke-sub-d3fba056-xdrrx\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.119222 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.119180 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:26.497013 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.496987 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx"] Oct 31 04:05:26.499864 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:05:26.499837 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c8bdb3_b40d_45cc_9165_9251f8a7612c.slice/crio-7ab2c33450a62a01ac9eadce9c9ea88e507a76a05060bae3b8427dd1b68108dc WatchSource:0}: Error finding container 7ab2c33450a62a01ac9eadce9c9ea88e507a76a05060bae3b8427dd1b68108dc: Status 404 returned error can't find the container with id 7ab2c33450a62a01ac9eadce9c9ea88e507a76a05060bae3b8427dd1b68108dc Oct 31 04:05:26.864757 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.864720 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" event={"ID":"78c8bdb3-b40d-45cc-9165-9251f8a7612c","Type":"ContainerStarted","Data":"4e7a893b46b0a26dfe1d8f4e895040810402b93bb69e46f677931e51d878790a"} Oct 31 04:05:26.865237 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.864779 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" event={"ID":"78c8bdb3-b40d-45cc-9165-9251f8a7612c","Type":"ContainerStarted","Data":"7ab2c33450a62a01ac9eadce9c9ea88e507a76a05060bae3b8427dd1b68108dc"} Oct 31 04:05:26.981109 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:26.980975 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" podStartSLOduration=1.9809585219999999 podStartE2EDuration="1.980958522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:26.979836951 +0000 UTC m=+2754.776555211" watchObservedRunningTime="2026-10-31 04:05:26.980958522 +0000 UTC m=+2754.777676795" Oct 31 04:05:27.871917 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:27.871870 4488 generic.go:356] "Generic (PLEG): container finished" podID="78c8bdb3-b40d-45cc-9165-9251f8a7612c" containerID="4e7a893b46b0a26dfe1d8f4e895040810402b93bb69e46f677931e51d878790a" exitCode=0 Oct 31 04:05:27.872382 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:27.871970 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" event={"ID":"78c8bdb3-b40d-45cc-9165-9251f8a7612c","Type":"ContainerDied","Data":"4e7a893b46b0a26dfe1d8f4e895040810402b93bb69e46f677931e51d878790a"} Oct 31 04:05:27.876826 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:27.876791 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:05:29.203772 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.203310 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:29.357820 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.355315 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca\") pod \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " Oct 31 04:05:29.357820 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.355401 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq\" (UniqueName: \"kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq\") pod \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " Oct 31 04:05:29.357820 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.355460 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca\") pod \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\" (UID: \"78c8bdb3-b40d-45cc-9165-9251f8a7612c\") " Oct 31 04:05:29.357820 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.357649 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca" pod "78c8bdb3-b40d-45cc-9165-9251f8a7612c" (UID: "78c8bdb3-b40d-45cc-9165-9251f8a7612c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:05:29.436195 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.434352 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq" pod "78c8bdb3-b40d-45cc-9165-9251f8a7612c" (UID: "78c8bdb3-b40d-45cc-9165-9251f8a7612c"). InnerVolumeSpecName "kube-api-access-hrggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:05:29.436413 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.436272 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca" pod "78c8bdb3-b40d-45cc-9165-9251f8a7612c" (UID: "78c8bdb3-b40d-45cc-9165-9251f8a7612c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:05:29.488199 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.469705 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:05:29.488199 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.469749 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrggq\" (UniqueName: \"kubernetes.io/projected/78c8bdb3-b40d-45cc-9165-9251f8a7612c-kube-api-access-hrggq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:05:29.488199 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.469768 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c8bdb3-b40d-45cc-9165-9251f8a7612c-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:05:29.906117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.905912 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" event={"ID":"78c8bdb3-b40d-45cc-9165-9251f8a7612c","Type":"ContainerDied","Data":"7ab2c33450a62a01ac9eadce9c9ea88e507a76a05060bae3b8427dd1b68108dc"} Oct 31 04:05:29.906117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.905982 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab2c33450a62a01ac9eadce9c9ea88e507a76a05060bae3b8427dd1b68108dc" Oct 31 04:05:29.906117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:29.906058 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx" Oct 31 04:05:31.053496 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:31.053456 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb"] Oct 31 04:05:31.102865 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:31.102819 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51456da3-p7bsb"] Oct 31 04:05:32.314778 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:32.314743 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088adc04-5b77-4390-8995-d645932f03ec" path="/var/lib/kubelet/pods/088adc04-5b77-4390-8995-d645932f03ec/volumes" Oct 31 04:05:32.811460 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:32.811422 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:05:35.569324 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.569288 4488 scope.go:122] "RemoveContainer" containerID="0d8679153d66ea4e9856d2b6881a792f894127a5fe05f57f638a4f2c231e400a" Oct 31 04:05:35.578528 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.578504 4488 scope.go:122] "RemoveContainer" containerID="f7ad3ce333eb69fc71ced6f8d903f22e2eee95134c7434859418cd432893459b" Oct 31 04:05:35.587530 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.587507 4488 scope.go:122] "RemoveContainer" containerID="c8f20ab07ede83dc186dfacbfa2126e05e1c0a1fd9e300b60a8acf1e9ecd57c6" Oct 31 04:05:35.596358 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.596326 4488 scope.go:122] "RemoveContainer" containerID="0b66cc12496bc01aa06bd1256be29eb3c992bfb3652d3723b3c266406c762893" Oct 31 04:05:35.605332 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.605308 4488 scope.go:122] "RemoveContainer" containerID="925ade31a42e7a409d7c0f0635d9e0075146a2b942d793d49ce59ac88faf3343" Oct 31 04:05:35.613984 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.613963 4488 scope.go:122] "RemoveContainer" containerID="b9c602a755da1b3e56eae483ba25bc75e39eba12deed53b70a0a86c4131460a0" Oct 31 04:05:35.622566 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:35.622546 4488 scope.go:122] "RemoveContainer" containerID="ebe63de2ea5f158835ff81cc297e3781756178f6e5ae720a2e84efce0ff47846" Oct 31 04:05:36.589197 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:05:36.589162 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:05:38.578876 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:38.578837 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:05:51.060344 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:05:51.060294 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:06:00.732300 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.732265 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg"] Oct 31 04:06:00.732714 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.732642 4488 cpu_manager.go:414] "containerMap: removing container" podUID="78c8bdb3-b40d-45cc-9165-9251f8a7612c" containerName="revoke-keys" Oct 31 04:06:00.732714 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.732653 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c8bdb3-b40d-45cc-9165-9251f8a7612c" containerName="revoke-keys" Oct 31 04:06:00.732796 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.732725 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c8bdb3-b40d-45cc-9165-9251f8a7612c" containerName="revoke-keys" Oct 31 04:06:00.737598 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.737579 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.841883 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.841844 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:00.842139 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.842012 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:00.842139 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.842120 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:00.842139 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.842123 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:06:00.843845 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.843825 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg"] Oct 31 04:06:00.880254 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.880211 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.880442 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.880282 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445kj\" (UniqueName: \"kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.880442 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.880382 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.981853 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.981820 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.982056 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.981879 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445kj\" (UniqueName: \"kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.982056 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.981926 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.982645 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.982591 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:00.984666 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:00.984635 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:01.044001 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:01.043961 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445kj\" (UniqueName: \"kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj\") pod \"maas-api-revoke-sub-ae594979-v4wlg\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:01.050760 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:01.050716 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:01.425864 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:01.425837 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg"] Oct 31 04:06:01.427946 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:06:01.427913 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cbf0c8_8c50_4510_9df6_97c301949712.slice/crio-b25b416d25798e8ac0ee4531b9d85c9c5d0d8ef8427972070a1da10db46db339 WatchSource:0}: Error finding container b25b416d25798e8ac0ee4531b9d85c9c5d0d8ef8427972070a1da10db46db339: Status 404 returned error can't find the container with id b25b416d25798e8ac0ee4531b9d85c9c5d0d8ef8427972070a1da10db46db339 Oct 31 04:06:02.136707 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:02.136672 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" event={"ID":"17cbf0c8-8c50-4510-9df6-97c301949712","Type":"ContainerStarted","Data":"89cb611773f9b251237f3ccca74dbf55adaa05fcd78002aef7dceb2283b5594b"} Oct 31 04:06:02.137153 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:02.136731 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" event={"ID":"17cbf0c8-8c50-4510-9df6-97c301949712","Type":"ContainerStarted","Data":"b25b416d25798e8ac0ee4531b9d85c9c5d0d8ef8427972070a1da10db46db339"} Oct 31 04:06:02.255003 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:02.254944 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" podStartSLOduration=2.254927845 podStartE2EDuration="2.254927845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:02.253917236 +0000 UTC m=+2790.050635495" watchObservedRunningTime="2026-10-31 04:06:02.254927845 +0000 UTC m=+2790.051646108" Oct 31 04:06:03.143184 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:03.143145 4488 generic.go:356] "Generic (PLEG): container finished" podID="17cbf0c8-8c50-4510-9df6-97c301949712" containerID="89cb611773f9b251237f3ccca74dbf55adaa05fcd78002aef7dceb2283b5594b" exitCode=0 Oct 31 04:06:03.143592 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:03.143236 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" event={"ID":"17cbf0c8-8c50-4510-9df6-97c301949712","Type":"ContainerDied","Data":"89cb611773f9b251237f3ccca74dbf55adaa05fcd78002aef7dceb2283b5594b"} Oct 31 04:06:04.292926 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.292899 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:04.416824 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.416731 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca\") pod \"17cbf0c8-8c50-4510-9df6-97c301949712\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " Oct 31 04:06:04.417015 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.416829 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca\") pod \"17cbf0c8-8c50-4510-9df6-97c301949712\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " Oct 31 04:06:04.417015 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.416898 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj\" (UniqueName: \"kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj\") pod \"17cbf0c8-8c50-4510-9df6-97c301949712\" (UID: \"17cbf0c8-8c50-4510-9df6-97c301949712\") " Oct 31 04:06:04.417172 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.417153 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca" pod "17cbf0c8-8c50-4510-9df6-97c301949712" (UID: "17cbf0c8-8c50-4510-9df6-97c301949712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:04.419439 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.419405 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca" pod "17cbf0c8-8c50-4510-9df6-97c301949712" (UID: "17cbf0c8-8c50-4510-9df6-97c301949712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:04.419777 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.419748 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj" pod "17cbf0c8-8c50-4510-9df6-97c301949712" (UID: "17cbf0c8-8c50-4510-9df6-97c301949712"). InnerVolumeSpecName "kube-api-access-445kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:04.517983 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.517943 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.517983 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.517977 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17cbf0c8-8c50-4510-9df6-97c301949712-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:06:04.517983 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:04.517988 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445kj\" (UniqueName: \"kubernetes.io/projected/17cbf0c8-8c50-4510-9df6-97c301949712-kube-api-access-445kj\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:06:05.154132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:05.154076 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" event={"ID":"17cbf0c8-8c50-4510-9df6-97c301949712","Type":"ContainerDied","Data":"b25b416d25798e8ac0ee4531b9d85c9c5d0d8ef8427972070a1da10db46db339"} Oct 31 04:06:05.154132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:05.154123 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg" Oct 31 04:06:05.154132 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:05.154141 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b25b416d25798e8ac0ee4531b9d85c9c5d0d8ef8427972070a1da10db46db339" Oct 31 04:06:06.589619 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:06:06.589578 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:06:13.590060 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:13.590022 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:06:23.289978 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:23.289935 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:06:28.907808 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.907773 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw"] Oct 31 04:06:28.908266 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.908192 4488 cpu_manager.go:414] "containerMap: removing container" podUID="17cbf0c8-8c50-4510-9df6-97c301949712" containerName="revoke-keys" Oct 31 04:06:28.908266 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.908205 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cbf0c8-8c50-4510-9df6-97c301949712" containerName="revoke-keys" Oct 31 04:06:28.908364 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.908282 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cbf0c8-8c50-4510-9df6-97c301949712" containerName="revoke-keys" Oct 31 04:06:28.912718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.912681 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:28.966784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.966666 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:28.986156 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.986121 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:28.986345 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.986177 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjh8\" (UniqueName: \"kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:28.986345 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:28.986294 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.016452 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.016408 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:29.016630 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.016456 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:29.016693 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.016673 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:06:29.018611 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.018587 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw"] Oct 31 04:06:29.087021 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.086985 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.087277 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.087034 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjh8\" (UniqueName: \"kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.087277 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.087066 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.087743 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.087715 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.089784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.089761 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.148824 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.148783 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjh8\" (UniqueName: \"kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8\") pod \"maas-api-revoke-sub-a4aef7d7-82wrw\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.231316 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.231233 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:29.609355 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:29.609327 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw"] Oct 31 04:06:29.611851 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:06:29.611804 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcc2445_b145_45a1_91df_bb1c768b86c6.slice/crio-94509eb9b8a05f0f656b057aa5f40a3cb32e7c37faa1f5344b7c6fc4fb9846a3 WatchSource:0}: Error finding container 94509eb9b8a05f0f656b057aa5f40a3cb32e7c37faa1f5344b7c6fc4fb9846a3: Status 404 returned error can't find the container with id 94509eb9b8a05f0f656b057aa5f40a3cb32e7c37faa1f5344b7c6fc4fb9846a3 Oct 31 04:06:30.291213 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:30.291122 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" event={"ID":"9bcc2445-b145-45a1-91df-bb1c768b86c6","Type":"ContainerStarted","Data":"67254708ba4d71255fd0e9216d6b0dd14cdd7271fa80a18e8c08e68178548cc7"} Oct 31 04:06:30.291213 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:30.291182 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" event={"ID":"9bcc2445-b145-45a1-91df-bb1c768b86c6","Type":"ContainerStarted","Data":"94509eb9b8a05f0f656b057aa5f40a3cb32e7c37faa1f5344b7c6fc4fb9846a3"} Oct 31 04:06:30.408358 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:30.408296 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" podStartSLOduration=2.408276792 podStartE2EDuration="2.408276792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:30.407319907 +0000 UTC m=+2818.204038167" watchObservedRunningTime="2026-10-31 04:06:30.408276792 +0000 UTC m=+2818.204995053" Oct 31 04:06:31.298068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:31.298029 4488 generic.go:356] "Generic (PLEG): container finished" podID="9bcc2445-b145-45a1-91df-bb1c768b86c6" containerID="67254708ba4d71255fd0e9216d6b0dd14cdd7271fa80a18e8c08e68178548cc7" exitCode=0 Oct 31 04:06:31.298521 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:31.298108 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" event={"ID":"9bcc2445-b145-45a1-91df-bb1c768b86c6","Type":"ContainerDied","Data":"67254708ba4d71255fd0e9216d6b0dd14cdd7271fa80a18e8c08e68178548cc7"} Oct 31 04:06:32.440686 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.440659 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:32.528862 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.528828 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8\" (UniqueName: \"kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8\") pod \"9bcc2445-b145-45a1-91df-bb1c768b86c6\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " Oct 31 04:06:32.529078 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.528919 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca\") pod \"9bcc2445-b145-45a1-91df-bb1c768b86c6\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " Oct 31 04:06:32.529189 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.529080 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca\") pod \"9bcc2445-b145-45a1-91df-bb1c768b86c6\" (UID: \"9bcc2445-b145-45a1-91df-bb1c768b86c6\") " Oct 31 04:06:32.529455 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.529427 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca" pod "9bcc2445-b145-45a1-91df-bb1c768b86c6" (UID: "9bcc2445-b145-45a1-91df-bb1c768b86c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:32.531307 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.531285 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca" pod "9bcc2445-b145-45a1-91df-bb1c768b86c6" (UID: "9bcc2445-b145-45a1-91df-bb1c768b86c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:32.531391 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.531284 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8" pod "9bcc2445-b145-45a1-91df-bb1c768b86c6" (UID: "9bcc2445-b145-45a1-91df-bb1c768b86c6"). InnerVolumeSpecName "kube-api-access-bzjh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:32.630260 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.630219 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzjh8\" (UniqueName: \"kubernetes.io/projected/9bcc2445-b145-45a1-91df-bb1c768b86c6-kube-api-access-bzjh8\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:06:32.630260 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.630255 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:06:32.630260 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:32.630264 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bcc2445-b145-45a1-91df-bb1c768b86c6-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:06:33.161264 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:33.161221 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:33.311200 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:33.311168 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" event={"ID":"9bcc2445-b145-45a1-91df-bb1c768b86c6","Type":"ContainerDied","Data":"94509eb9b8a05f0f656b057aa5f40a3cb32e7c37faa1f5344b7c6fc4fb9846a3"} Oct 31 04:06:33.311395 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:33.311211 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw" Oct 31 04:06:33.311395 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:33.311214 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94509eb9b8a05f0f656b057aa5f40a3cb32e7c37faa1f5344b7c6fc4fb9846a3" Oct 31 04:06:35.672622 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:35.672587 4488 scope.go:122] "RemoveContainer" containerID="bbae2834008fec825bbb7f17221ff76616de8af316a1a6fe8acde1769120a59d" Oct 31 04:06:35.681675 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:35.681649 4488 scope.go:122] "RemoveContainer" containerID="d0f37e978ad07c11b564fece391c44c8130d04639a505660cf3df9c3e97f9971" Oct 31 04:06:35.689844 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:35.689824 4488 scope.go:122] "RemoveContainer" containerID="ee4ba3c118168399423edb544dfe7f6a105bb5ad11139a8d1dc012faeb244fd3" Oct 31 04:06:35.698481 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:35.698455 4488 scope.go:122] "RemoveContainer" containerID="bb48a82c602ed2fc3224570d359169e2445e0b5d91433bf35239b0eee66c46ab" Oct 31 04:06:36.588856 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:06:36.588822 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:06:41.803633 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:41.803593 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:06:51.070118 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:51.069673 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl"] Oct 31 04:06:51.129394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:51.129329 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89be5563-7mbrl"] Oct 31 04:06:52.315571 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:52.315539 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa741add-ff69-481e-888d-914f3dfb72a5" path="/var/lib/kubelet/pods/aa741add-ff69-481e-888d-914f3dfb72a5/volumes" Oct 31 04:06:53.065375 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:53.065319 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:54.476190 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:54.476160 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:06:59.713490 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.713453 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln"] Oct 31 04:06:59.713909 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.713842 4488 cpu_manager.go:414] "containerMap: removing container" podUID="9bcc2445-b145-45a1-91df-bb1c768b86c6" containerName="revoke-keys" Oct 31 04:06:59.713909 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.713854 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcc2445-b145-45a1-91df-bb1c768b86c6" containerName="revoke-keys" Oct 31 04:06:59.714788 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.714752 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcc2445-b145-45a1-91df-bb1c768b86c6" containerName="revoke-keys" Oct 31 04:06:59.719949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.719912 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.773947 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.773897 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:59.774317 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.774143 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:06:59.774437 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.774161 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:59.774437 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.774187 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:59.776569 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.776539 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln"] Oct 31 04:06:59.825140 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.825100 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656jk\" (UniqueName: \"kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.825140 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.825144 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.825378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.825261 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.926446 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.926403 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656jk\" (UniqueName: \"kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.926446 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.926451 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.926679 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.926507 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.927142 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.927120 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.929017 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.928993 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:06:59.988231 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:06:59.988153 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-656jk\" (UniqueName: \"kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk\") pod \"maas-api-revoke-sub-79805ca9-9lmln\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:07:00.034284 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:00.034244 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:07:00.408660 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:00.408615 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln"] Oct 31 04:07:00.411035 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:07:00.411008 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2ce102_b329_425c_96a6_b9dcedd8805e.slice/crio-198a3d2d1509488398425dd70bb939bf12d31ec2b2bffd8489e0ea4cd09a123b WatchSource:0}: Error finding container 198a3d2d1509488398425dd70bb939bf12d31ec2b2bffd8489e0ea4cd09a123b: Status 404 returned error can't find the container with id 198a3d2d1509488398425dd70bb939bf12d31ec2b2bffd8489e0ea4cd09a123b Oct 31 04:07:00.470370 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:00.470335 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" event={"ID":"ec2ce102-b329-425c-96a6-b9dcedd8805e","Type":"ContainerStarted","Data":"198a3d2d1509488398425dd70bb939bf12d31ec2b2bffd8489e0ea4cd09a123b"} Oct 31 04:07:01.476765 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:01.476738 4488 generic.go:356] "Generic (PLEG): container finished" podID="ec2ce102-b329-425c-96a6-b9dcedd8805e" containerID="7ee5e15d6abf4182c7e966fda3b22f250ad49f70f5ea7fd36625cccce61e63de" exitCode=0 Oct 31 04:07:01.477213 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:01.476834 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" event={"ID":"ec2ce102-b329-425c-96a6-b9dcedd8805e","Type":"ContainerDied","Data":"7ee5e15d6abf4182c7e966fda3b22f250ad49f70f5ea7fd36625cccce61e63de"} Oct 31 04:07:02.621106 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.621069 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:07:02.652026 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.651984 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca\") pod \"ec2ce102-b329-425c-96a6-b9dcedd8805e\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " Oct 31 04:07:02.652026 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.652028 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk\" (UniqueName: \"kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk\") pod \"ec2ce102-b329-425c-96a6-b9dcedd8805e\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " Oct 31 04:07:02.652320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.652084 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca\") pod \"ec2ce102-b329-425c-96a6-b9dcedd8805e\" (UID: \"ec2ce102-b329-425c-96a6-b9dcedd8805e\") " Oct 31 04:07:02.652539 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.652508 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca" pod "ec2ce102-b329-425c-96a6-b9dcedd8805e" (UID: "ec2ce102-b329-425c-96a6-b9dcedd8805e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:02.654452 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.654421 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca" pod "ec2ce102-b329-425c-96a6-b9dcedd8805e" (UID: "ec2ce102-b329-425c-96a6-b9dcedd8805e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:02.654554 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.654456 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk" pod "ec2ce102-b329-425c-96a6-b9dcedd8805e" (UID: "ec2ce102-b329-425c-96a6-b9dcedd8805e"). InnerVolumeSpecName "kube-api-access-656jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:02.753406 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.753306 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:02.753406 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.753342 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-656jk\" (UniqueName: \"kubernetes.io/projected/ec2ce102-b329-425c-96a6-b9dcedd8805e-kube-api-access-656jk\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:02.753406 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:02.753357 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2ce102-b329-425c-96a6-b9dcedd8805e-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:03.488560 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:03.488528 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" Oct 31 04:07:03.488560 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:03.488548 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln" event={"ID":"ec2ce102-b329-425c-96a6-b9dcedd8805e","Type":"ContainerDied","Data":"198a3d2d1509488398425dd70bb939bf12d31ec2b2bffd8489e0ea4cd09a123b"} Oct 31 04:07:03.488813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:03.488597 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="198a3d2d1509488398425dd70bb939bf12d31ec2b2bffd8489e0ea4cd09a123b" Oct 31 04:07:06.588686 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:07:06.588651 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:07:14.394876 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:14.394835 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:07:23.603408 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.603368 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg"] Oct 31 04:07:23.603989 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.603966 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ec2ce102-b329-425c-96a6-b9dcedd8805e" containerName="revoke-keys" Oct 31 04:07:23.604116 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.603992 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2ce102-b329-425c-96a6-b9dcedd8805e" containerName="revoke-keys" Oct 31 04:07:23.604196 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.604114 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2ce102-b329-425c-96a6-b9dcedd8805e" containerName="revoke-keys" Oct 31 04:07:23.607438 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.607415 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.662847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.662815 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:07:23.678868 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.678821 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5945s\" (UniqueName: \"kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.679083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.678878 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.679083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.679037 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.712371 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.712331 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:07:23.712371 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.712329 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:07:23.712629 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.712519 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:07:23.714541 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.714517 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg"] Oct 31 04:07:23.780530 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.780487 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.780751 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.780553 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.780751 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.780627 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5945s\" (UniqueName: \"kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.781256 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.781226 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.783136 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.783114 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.842614 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.842579 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5945s\" (UniqueName: \"kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s\") pod \"maas-api-revoke-sub-0b7e8edb-sf5wg\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:23.921317 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:23.921218 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:24.297932 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:24.297901 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg"] Oct 31 04:07:24.300144 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:07:24.300115 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb926d8d0_47f9_4f8f_a7bc_4ca0f8657937.slice/crio-65cee1a6c4e0e4da993bd6df6c0ca44f6e2b038a793a30e143681b5414e25163 WatchSource:0}: Error finding container 65cee1a6c4e0e4da993bd6df6c0ca44f6e2b038a793a30e143681b5414e25163: Status 404 returned error can't find the container with id 65cee1a6c4e0e4da993bd6df6c0ca44f6e2b038a793a30e143681b5414e25163 Oct 31 04:07:24.620878 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:24.620845 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" event={"ID":"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937","Type":"ContainerStarted","Data":"65cee1a6c4e0e4da993bd6df6c0ca44f6e2b038a793a30e143681b5414e25163"} Oct 31 04:07:25.627805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:25.627770 4488 generic.go:356] "Generic (PLEG): container finished" podID="b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" containerID="95ca8a9138248d98fa334d22960ce213d449ef142cdb73a76ae564289a608ac1" exitCode=0 Oct 31 04:07:25.628321 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:25.627856 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" event={"ID":"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937","Type":"ContainerDied","Data":"95ca8a9138248d98fa334d22960ce213d449ef142cdb73a76ae564289a608ac1"} Oct 31 04:07:26.769579 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.769553 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:26.914389 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.914281 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s\" (UniqueName: \"kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s\") pod \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " Oct 31 04:07:26.914389 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.914349 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca\") pod \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " Oct 31 04:07:26.914389 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.914369 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca\") pod \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\" (UID: \"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937\") " Oct 31 04:07:26.914744 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.914717 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca" pod "b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" (UID: "b926d8d0-47f9-4f8f-a7bc-4ca0f8657937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:26.916805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.916772 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca" pod "b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" (UID: "b926d8d0-47f9-4f8f-a7bc-4ca0f8657937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:26.916805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:26.916773 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s" pod "b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" (UID: "b926d8d0-47f9-4f8f-a7bc-4ca0f8657937"). InnerVolumeSpecName "kube-api-access-5945s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:27.015627 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:27.015592 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5945s\" (UniqueName: \"kubernetes.io/projected/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-kube-api-access-5945s\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:27.015627 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:27.015622 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:27.015627 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:27.015634 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:27.639105 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:27.639058 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" Oct 31 04:07:27.639105 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:27.639050 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg" event={"ID":"b926d8d0-47f9-4f8f-a7bc-4ca0f8657937","Type":"ContainerDied","Data":"65cee1a6c4e0e4da993bd6df6c0ca44f6e2b038a793a30e143681b5414e25163"} Oct 31 04:07:27.639342 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:27.639151 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65cee1a6c4e0e4da993bd6df6c0ca44f6e2b038a793a30e143681b5414e25163" Oct 31 04:07:33.511580 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:33.511537 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:34.974596 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:34.974558 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn"] Oct 31 04:07:34.975034 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:34.974958 4488 cpu_manager.go:414] "containerMap: removing container" podUID="b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" containerName="revoke-keys" Oct 31 04:07:34.975034 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:34.974972 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" containerName="revoke-keys" Oct 31 04:07:34.975147 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:34.975043 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" containerName="revoke-keys" Oct 31 04:07:34.978117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:34.978084 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.030857 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.030821 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:07:35.080874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.080834 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:07:35.080874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.080848 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:07:35.083188 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.083167 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn"] Oct 31 04:07:35.098243 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.098214 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.098347 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.098267 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.098347 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.098286 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqmk\" (UniqueName: \"kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.130791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.130760 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:07:35.199863 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.199817 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.199863 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.199870 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqmk\" (UniqueName: \"kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.200143 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.199954 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.200651 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.200628 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.202452 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.202425 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.261979 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.261901 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqmk\" (UniqueName: \"kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk\") pod \"maas-api-revoke-sub-3d4156f1-lgtcn\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.295952 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.295910 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:35.673513 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.673483 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn"] Oct 31 04:07:35.675892 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:07:35.675841 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33bb137_80f8_4d98_98f9_0834798896b5.slice/crio-e9bc3f98c6fd3d16996424ffbb57307e5a69fad17078d750f9240ce693d9019c WatchSource:0}: Error finding container e9bc3f98c6fd3d16996424ffbb57307e5a69fad17078d750f9240ce693d9019c: Status 404 returned error can't find the container with id e9bc3f98c6fd3d16996424ffbb57307e5a69fad17078d750f9240ce693d9019c Oct 31 04:07:35.731606 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.731573 4488 scope.go:122] "RemoveContainer" containerID="de2407a41b2dd8a274ef49547a99100f9f86f55a88a3c91fd24bb9f7a0b5d124" Oct 31 04:07:35.740703 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:35.740677 4488 scope.go:122] "RemoveContainer" containerID="e27fa04500ef2104834c3f51a297621c6a23277fa189698c4ddfde2cd17d984d" Oct 31 04:07:36.493751 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:36.493713 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:07:36.592684 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:07:36.592651 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:07:36.686066 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:36.686033 4488 generic.go:356] "Generic (PLEG): container finished" podID="f33bb137-80f8-4d98-98f9-0834798896b5" containerID="ce8d2c849d896a97863ef04b3c32c4fb52dd29739d2f8a6851b0a30b44f3a5bf" exitCode=0 Oct 31 04:07:36.686260 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:36.686126 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" event={"ID":"f33bb137-80f8-4d98-98f9-0834798896b5","Type":"ContainerDied","Data":"ce8d2c849d896a97863ef04b3c32c4fb52dd29739d2f8a6851b0a30b44f3a5bf"} Oct 31 04:07:36.686260 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:36.686182 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" event={"ID":"f33bb137-80f8-4d98-98f9-0834798896b5","Type":"ContainerStarted","Data":"e9bc3f98c6fd3d16996424ffbb57307e5a69fad17078d750f9240ce693d9019c"} Oct 31 04:07:38.432116 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.423435 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:38.468924 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.468866 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk\" (UniqueName: \"kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk\") pod \"f33bb137-80f8-4d98-98f9-0834798896b5\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " Oct 31 04:07:38.469361 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.469324 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca\") pod \"f33bb137-80f8-4d98-98f9-0834798896b5\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " Oct 31 04:07:38.470125 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.470080 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca\") pod \"f33bb137-80f8-4d98-98f9-0834798896b5\" (UID: \"f33bb137-80f8-4d98-98f9-0834798896b5\") " Oct 31 04:07:38.471141 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.471117 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca" pod "f33bb137-80f8-4d98-98f9-0834798896b5" (UID: "f33bb137-80f8-4d98-98f9-0834798896b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:38.474902 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.474862 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk" pod "f33bb137-80f8-4d98-98f9-0834798896b5" (UID: "f33bb137-80f8-4d98-98f9-0834798896b5"). InnerVolumeSpecName "kube-api-access-dvqmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:38.496927 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.496886 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca" pod "f33bb137-80f8-4d98-98f9-0834798896b5" (UID: "f33bb137-80f8-4d98-98f9-0834798896b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:07:38.590051 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.589948 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:38.590344 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.590319 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33bb137-80f8-4d98-98f9-0834798896b5-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:38.590455 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.590440 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqmk\" (UniqueName: \"kubernetes.io/projected/f33bb137-80f8-4d98-98f9-0834798896b5-kube-api-access-dvqmk\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:07:38.709875 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.709830 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" event={"ID":"f33bb137-80f8-4d98-98f9-0834798896b5","Type":"ContainerDied","Data":"e9bc3f98c6fd3d16996424ffbb57307e5a69fad17078d750f9240ce693d9019c"} Oct 31 04:07:38.710186 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.710163 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bc3f98c6fd3d16996424ffbb57307e5a69fad17078d750f9240ce693d9019c" Oct 31 04:07:38.710465 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:38.710449 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn" Oct 31 04:07:49.057803 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:49.057760 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb"] Oct 31 04:07:49.107724 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:49.107692 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ww8nb"] Oct 31 04:07:49.700159 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:49.700117 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:07:50.318862 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:07:50.318820 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb73361-563b-4996-aa37-669963e66b1d" path="/var/lib/kubelet/pods/3eb73361-563b-4996-aa37-669963e66b1d/volumes" Oct 31 04:08:00.613068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:00.613034 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:08:06.588905 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:08:06.588863 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:08:08.827543 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.827505 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn"] Oct 31 04:08:08.877244 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.877207 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lgtcn"] Oct 31 04:08:08.960869 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.960830 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g"] Oct 31 04:08:08.961277 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.961263 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f33bb137-80f8-4d98-98f9-0834798896b5" containerName="revoke-keys" Oct 31 04:08:08.961349 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.961280 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33bb137-80f8-4d98-98f9-0834798896b5" containerName="revoke-keys" Oct 31 04:08:08.961406 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.961356 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33bb137-80f8-4d98-98f9-0834798896b5" containerName="revoke-keys" Oct 31 04:08:08.964271 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:08.964254 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.045942 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.045907 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:08:09.053473 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.053432 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.053682 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.053485 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.053682 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.053603 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2jr\" (UniqueName: \"kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.067474 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.067437 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g"] Oct 31 04:08:09.074428 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.074399 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:08:09.074600 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.074462 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:08:09.146293 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.146205 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:08:09.154696 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.154655 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.154860 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.154707 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.154860 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.154749 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2jr\" (UniqueName: \"kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.155581 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.155499 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.157336 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.157310 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.216583 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.216542 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2jr\" (UniqueName: \"kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr\") pod \"maas-api-revoke-sub-3d4156f1-d5f6g\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.278462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.278415 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:09.656554 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.656521 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g"] Oct 31 04:08:09.659893 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:08:09.659850 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575f4e17_98a2_4d92_97c0_959f06c588ca.slice/crio-3fb779a5767a3f3bdf4ca9bd5bc1084b31a81d6c783972ed47259c243abeddbf WatchSource:0}: Error finding container 3fb779a5767a3f3bdf4ca9bd5bc1084b31a81d6c783972ed47259c243abeddbf: Status 404 returned error can't find the container with id 3fb779a5767a3f3bdf4ca9bd5bc1084b31a81d6c783972ed47259c243abeddbf Oct 31 04:08:09.907710 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:09.907623 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" event={"ID":"575f4e17-98a2-4d92-97c0-959f06c588ca","Type":"ContainerStarted","Data":"3fb779a5767a3f3bdf4ca9bd5bc1084b31a81d6c783972ed47259c243abeddbf"} Oct 31 04:08:10.315818 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:10.315738 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33bb137-80f8-4d98-98f9-0834798896b5" path="/var/lib/kubelet/pods/f33bb137-80f8-4d98-98f9-0834798896b5/volumes" Oct 31 04:08:10.914097 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:10.914058 4488 generic.go:356] "Generic (PLEG): container finished" podID="575f4e17-98a2-4d92-97c0-959f06c588ca" containerID="7ac892530b97ac8863663474906bd87d4867b4561c9c1db425a1f131a3e81bf3" exitCode=0 Oct 31 04:08:10.914522 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:10.914149 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" event={"ID":"575f4e17-98a2-4d92-97c0-959f06c588ca","Type":"ContainerDied","Data":"7ac892530b97ac8863663474906bd87d4867b4561c9c1db425a1f131a3e81bf3"} Oct 31 04:08:12.059628 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.059600 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:12.185166 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.185068 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr\" (UniqueName: \"kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr\") pod \"575f4e17-98a2-4d92-97c0-959f06c588ca\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " Oct 31 04:08:12.185335 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.185241 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca\") pod \"575f4e17-98a2-4d92-97c0-959f06c588ca\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " Oct 31 04:08:12.185335 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.185291 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca\") pod \"575f4e17-98a2-4d92-97c0-959f06c588ca\" (UID: \"575f4e17-98a2-4d92-97c0-959f06c588ca\") " Oct 31 04:08:12.185805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.185776 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca" pod "575f4e17-98a2-4d92-97c0-959f06c588ca" (UID: "575f4e17-98a2-4d92-97c0-959f06c588ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:12.187848 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.187812 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr" pod "575f4e17-98a2-4d92-97c0-959f06c588ca" (UID: "575f4e17-98a2-4d92-97c0-959f06c588ca"). InnerVolumeSpecName "kube-api-access-dd2jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:12.188019 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.187922 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca" pod "575f4e17-98a2-4d92-97c0-959f06c588ca" (UID: "575f4e17-98a2-4d92-97c0-959f06c588ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:08:12.287274 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.287235 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2jr\" (UniqueName: \"kubernetes.io/projected/575f4e17-98a2-4d92-97c0-959f06c588ca-kube-api-access-dd2jr\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:08:12.287274 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.287267 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:08:12.287274 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.287277 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575f4e17-98a2-4d92-97c0-959f06c588ca-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:08:12.925876 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.925846 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" Oct 31 04:08:12.926051 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.925844 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g" event={"ID":"575f4e17-98a2-4d92-97c0-959f06c588ca","Type":"ContainerDied","Data":"3fb779a5767a3f3bdf4ca9bd5bc1084b31a81d6c783972ed47259c243abeddbf"} Oct 31 04:08:12.926051 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:12.925972 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fb779a5767a3f3bdf4ca9bd5bc1084b31a81d6c783972ed47259c243abeddbf" Oct 31 04:08:23.579736 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:23.579648 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:08:32.931330 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:32.931282 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:08:33.782043 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:33.782002 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:08:33.855734 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:33.855692 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:08:35.765698 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:35.765667 4488 scope.go:122] "RemoveContainer" containerID="458038ab37ab1a9f88695b6c0dc5bbb970dd461704ddb1fe7d54cce837fdae3c" Oct 31 04:08:36.589534 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:08:36.589491 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:08:42.705542 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:42.705507 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:08:47.607153 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:47.607114 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:08:56.500642 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.500604 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh"] Oct 31 04:08:56.501068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.500997 4488 cpu_manager.go:414] "containerMap: removing container" podUID="575f4e17-98a2-4d92-97c0-959f06c588ca" containerName="revoke-keys" Oct 31 04:08:56.501068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.501009 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575f4e17-98a2-4d92-97c0-959f06c588ca" containerName="revoke-keys" Oct 31 04:08:56.501165 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.501138 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="575f4e17-98a2-4d92-97c0-959f06c588ca" containerName="revoke-keys" Oct 31 04:08:56.505560 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.505520 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.558118 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.558050 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:08:56.558296 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.558175 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:08:56.608070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.608024 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:08:56.608293 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.608175 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:08:56.610411 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.610387 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh"] Oct 31 04:08:56.647649 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.647611 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.647846 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.647676 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.647846 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.647781 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbt2\" (UniqueName: \"kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.748730 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.748690 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.748938 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.748760 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.748938 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.748836 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbt2\" (UniqueName: \"kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.749500 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.749479 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.751378 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.751335 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.810638 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.810595 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbt2\" (UniqueName: \"kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2\") pod \"maas-api-revoke-sub-e2fc9c5b-5f4wh\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:56.819251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:56.819220 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:57.195607 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:57.195581 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh"] Oct 31 04:08:57.198117 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:08:57.198060 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5339dd_2599_42ce_9179_2af9a5264f1d.slice/crio-3068268e97d1943ba2d2e72133bf0084bbf302b2feb4700f86478ed99e07122a WatchSource:0}: Error finding container 3068268e97d1943ba2d2e72133bf0084bbf302b2feb4700f86478ed99e07122a: Status 404 returned error can't find the container with id 3068268e97d1943ba2d2e72133bf0084bbf302b2feb4700f86478ed99e07122a Oct 31 04:08:58.184197 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:58.184150 4488 generic.go:356] "Generic (PLEG): container finished" podID="ec5339dd-2599-42ce-9179-2af9a5264f1d" containerID="ec8b3b9f99d455b33833b603a8244cba906714ee65ebfbb636a55639df1445f6" exitCode=0 Oct 31 04:08:58.184652 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:58.184233 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" event={"ID":"ec5339dd-2599-42ce-9179-2af9a5264f1d","Type":"ContainerDied","Data":"ec8b3b9f99d455b33833b603a8244cba906714ee65ebfbb636a55639df1445f6"} Oct 31 04:08:58.184652 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:58.184316 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" event={"ID":"ec5339dd-2599-42ce-9179-2af9a5264f1d","Type":"ContainerStarted","Data":"3068268e97d1943ba2d2e72133bf0084bbf302b2feb4700f86478ed99e07122a"} Oct 31 04:08:59.341654 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.341628 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:08:59.476342 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.476252 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca\") pod \"ec5339dd-2599-42ce-9179-2af9a5264f1d\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " Oct 31 04:08:59.476342 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.476301 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca\") pod \"ec5339dd-2599-42ce-9179-2af9a5264f1d\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " Oct 31 04:08:59.476606 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.476394 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2\" (UniqueName: \"kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2\") pod \"ec5339dd-2599-42ce-9179-2af9a5264f1d\" (UID: \"ec5339dd-2599-42ce-9179-2af9a5264f1d\") " Oct 31 04:08:59.476768 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.476727 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca" pod "ec5339dd-2599-42ce-9179-2af9a5264f1d" (UID: "ec5339dd-2599-42ce-9179-2af9a5264f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:59.478791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.478760 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca" pod "ec5339dd-2599-42ce-9179-2af9a5264f1d" (UID: "ec5339dd-2599-42ce-9179-2af9a5264f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:08:59.478791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.478780 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2" pod "ec5339dd-2599-42ce-9179-2af9a5264f1d" (UID: "ec5339dd-2599-42ce-9179-2af9a5264f1d"). InnerVolumeSpecName "kube-api-access-rpbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:59.578330 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.578288 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbt2\" (UniqueName: \"kubernetes.io/projected/ec5339dd-2599-42ce-9179-2af9a5264f1d-kube-api-access-rpbt2\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:08:59.578330 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.578323 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:08:59.578330 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:08:59.578333 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5339dd-2599-42ce-9179-2af9a5264f1d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:00.051823 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:00.051786 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j"] Oct 31 04:09:00.055016 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:00.054992 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qz95j"] Oct 31 04:09:00.197126 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:00.197098 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" Oct 31 04:09:00.197286 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:00.197077 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh" event={"ID":"ec5339dd-2599-42ce-9179-2af9a5264f1d","Type":"ContainerDied","Data":"3068268e97d1943ba2d2e72133bf0084bbf302b2feb4700f86478ed99e07122a"} Oct 31 04:09:00.197286 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:00.197220 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3068268e97d1943ba2d2e72133bf0084bbf302b2feb4700f86478ed99e07122a" Oct 31 04:09:00.315709 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:00.315630 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7974b239-bc52-407d-aec6-bcd8b1cc3471" path="/var/lib/kubelet/pods/7974b239-bc52-407d-aec6-bcd8b1cc3471/volumes" Oct 31 04:09:06.589102 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:09:06.589047 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:09:07.303416 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.303379 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl"] Oct 31 04:09:07.303763 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.303751 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ec5339dd-2599-42ce-9179-2af9a5264f1d" containerName="revoke-keys" Oct 31 04:09:07.303810 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.303765 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5339dd-2599-42ce-9179-2af9a5264f1d" containerName="revoke-keys" Oct 31 04:09:07.303879 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.303870 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5339dd-2599-42ce-9179-2af9a5264f1d" containerName="revoke-keys" Oct 31 04:09:07.306916 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.306899 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.359697 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.359657 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:07.362357 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.362337 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:07.409484 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.409451 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:09:07.411657 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.411637 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl"] Oct 31 04:09:07.455003 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.454967 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.455236 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.455010 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lzxw\" (UniqueName: \"kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.455236 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.455040 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.460120 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.460075 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:07.556803 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.556706 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.556974 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.556822 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.556974 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.556846 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lzxw\" (UniqueName: \"kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.557558 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.557540 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.559508 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.559488 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.618533 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.618501 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lzxw\" (UniqueName: \"kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw\") pod \"maas-api-revoke-sub-5a29efb3-sv2cl\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.621336 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.621312 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:07.997450 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:07.997425 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl"] Oct 31 04:09:08.000214 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:09:08.000185 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ad9320_e3d1_45fe_87f0_a63040fb6fa7.slice/crio-acac85e64ddd5d47f3454c52a7ba2f39e8181982b97f75e696c877dde1887fb6 WatchSource:0}: Error finding container acac85e64ddd5d47f3454c52a7ba2f39e8181982b97f75e696c877dde1887fb6: Status 404 returned error can't find the container with id acac85e64ddd5d47f3454c52a7ba2f39e8181982b97f75e696c877dde1887fb6 Oct 31 04:09:08.245158 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:08.245122 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" event={"ID":"43ad9320-e3d1-45fe-87f0-a63040fb6fa7","Type":"ContainerStarted","Data":"acac85e64ddd5d47f3454c52a7ba2f39e8181982b97f75e696c877dde1887fb6"} Oct 31 04:09:09.251246 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:09.251158 4488 generic.go:356] "Generic (PLEG): container finished" podID="43ad9320-e3d1-45fe-87f0-a63040fb6fa7" containerID="dac356b883765e5b9f822a2b1f407ca76c0d2cf1f3d344a4910c731a85312f73" exitCode=0 Oct 31 04:09:09.251246 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:09.251210 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" event={"ID":"43ad9320-e3d1-45fe-87f0-a63040fb6fa7","Type":"ContainerDied","Data":"dac356b883765e5b9f822a2b1f407ca76c0d2cf1f3d344a4910c731a85312f73"} Oct 31 04:09:09.469930 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:09.469892 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:09:10.465179 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.465130 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:10.590127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.587145 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw\" (UniqueName: \"kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw\") pod \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " Oct 31 04:09:10.590127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.587233 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca\") pod \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " Oct 31 04:09:10.590127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.587269 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca\") pod \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\" (UID: \"43ad9320-e3d1-45fe-87f0-a63040fb6fa7\") " Oct 31 04:09:10.590127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.588875 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca" pod "43ad9320-e3d1-45fe-87f0-a63040fb6fa7" (UID: "43ad9320-e3d1-45fe-87f0-a63040fb6fa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:10.590127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.589668 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:10.594759 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.594707 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca" pod "43ad9320-e3d1-45fe-87f0-a63040fb6fa7" (UID: "43ad9320-e3d1-45fe-87f0-a63040fb6fa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:10.599742 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.599705 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw" pod "43ad9320-e3d1-45fe-87f0-a63040fb6fa7" (UID: "43ad9320-e3d1-45fe-87f0-a63040fb6fa7"). InnerVolumeSpecName "kube-api-access-2lzxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:10.690594 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.690540 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lzxw\" (UniqueName: \"kubernetes.io/projected/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-kube-api-access-2lzxw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:10.690594 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:10.690594 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ad9320-e3d1-45fe-87f0-a63040fb6fa7-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:11.376117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:11.374789 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" event={"ID":"43ad9320-e3d1-45fe-87f0-a63040fb6fa7","Type":"ContainerDied","Data":"acac85e64ddd5d47f3454c52a7ba2f39e8181982b97f75e696c877dde1887fb6"} Oct 31 04:09:11.376117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:11.374865 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acac85e64ddd5d47f3454c52a7ba2f39e8181982b97f75e696c877dde1887fb6" Oct 31 04:09:11.376117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:11.374979 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl" Oct 31 04:09:20.205871 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:20.205832 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:09:24.277415 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.277377 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g"] Oct 31 04:09:24.327679 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.327646 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5f6g"] Oct 31 04:09:24.397816 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.397783 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r"] Oct 31 04:09:24.398182 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.398167 4488 cpu_manager.go:414] "containerMap: removing container" podUID="43ad9320-e3d1-45fe-87f0-a63040fb6fa7" containerName="revoke-keys" Oct 31 04:09:24.398247 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.398183 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ad9320-e3d1-45fe-87f0-a63040fb6fa7" containerName="revoke-keys" Oct 31 04:09:24.398287 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.398275 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ad9320-e3d1-45fe-87f0-a63040fb6fa7" containerName="revoke-keys" Oct 31 04:09:24.401944 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.401922 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.456851 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.456823 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:24.457670 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.457319 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:09:24.462137 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.462081 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r"] Oct 31 04:09:24.473838 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.473806 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.474012 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.473932 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.474012 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.473972 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5wm\" (UniqueName: \"kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.534712 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.534618 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:24.534712 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.534694 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:24.575066 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.575028 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.575292 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.575119 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.575292 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.575146 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5wm\" (UniqueName: \"kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.575702 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.575678 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.577750 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.577723 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.636478 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.636444 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5wm\" (UniqueName: \"kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm\") pod \"maas-api-revoke-sub-3d4156f1-rf57r\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:24.715196 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:24.715155 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:25.094107 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:25.094059 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r"] Oct 31 04:09:25.096217 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:09:25.096187 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dc20f0_d11a_4f2e_becb_b7be97429801.slice/crio-8603a8266a65ec3c302bd5dd80866f23c086e298905e0ca83a377a5f1054c50b WatchSource:0}: Error finding container 8603a8266a65ec3c302bd5dd80866f23c086e298905e0ca83a377a5f1054c50b: Status 404 returned error can't find the container with id 8603a8266a65ec3c302bd5dd80866f23c086e298905e0ca83a377a5f1054c50b Oct 31 04:09:25.477214 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:25.477165 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" event={"ID":"58dc20f0-d11a-4f2e-becb-b7be97429801","Type":"ContainerStarted","Data":"42af9bdac56e1535b66f935b7c1699ab2ad5210878d12f5205b0954bc585dc09"} Oct 31 04:09:25.477620 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:25.477231 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" event={"ID":"58dc20f0-d11a-4f2e-becb-b7be97429801","Type":"ContainerStarted","Data":"8603a8266a65ec3c302bd5dd80866f23c086e298905e0ca83a377a5f1054c50b"} Oct 31 04:09:25.595307 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:25.595250 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" podStartSLOduration=1.5952347580000001 podStartE2EDuration="1.595234758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:25.593118519 +0000 UTC m=+2993.389836771" watchObservedRunningTime="2026-10-31 04:09:25.595234758 +0000 UTC m=+2993.391953017" Oct 31 04:09:26.315029 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:26.314945 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575f4e17-98a2-4d92-97c0-959f06c588ca" path="/var/lib/kubelet/pods/575f4e17-98a2-4d92-97c0-959f06c588ca/volumes" Oct 31 04:09:26.484669 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:26.484629 4488 generic.go:356] "Generic (PLEG): container finished" podID="58dc20f0-d11a-4f2e-becb-b7be97429801" containerID="42af9bdac56e1535b66f935b7c1699ab2ad5210878d12f5205b0954bc585dc09" exitCode=0 Oct 31 04:09:26.485122 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:26.484725 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" event={"ID":"58dc20f0-d11a-4f2e-becb-b7be97429801","Type":"ContainerDied","Data":"42af9bdac56e1535b66f935b7c1699ab2ad5210878d12f5205b0954bc585dc09"} Oct 31 04:09:27.628444 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.628416 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:27.708380 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.708342 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm\" (UniqueName: \"kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm\") pod \"58dc20f0-d11a-4f2e-becb-b7be97429801\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " Oct 31 04:09:27.708380 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.708391 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca\") pod \"58dc20f0-d11a-4f2e-becb-b7be97429801\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " Oct 31 04:09:27.708675 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.708448 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca\") pod \"58dc20f0-d11a-4f2e-becb-b7be97429801\" (UID: \"58dc20f0-d11a-4f2e-becb-b7be97429801\") " Oct 31 04:09:27.708847 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.708813 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca" pod "58dc20f0-d11a-4f2e-becb-b7be97429801" (UID: "58dc20f0-d11a-4f2e-becb-b7be97429801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:27.710737 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.710711 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm" pod "58dc20f0-d11a-4f2e-becb-b7be97429801" (UID: "58dc20f0-d11a-4f2e-becb-b7be97429801"). InnerVolumeSpecName "kube-api-access-cq5wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:27.710849 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.710804 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca" pod "58dc20f0-d11a-4f2e-becb-b7be97429801" (UID: "58dc20f0-d11a-4f2e-becb-b7be97429801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:27.809592 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.809490 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5wm\" (UniqueName: \"kubernetes.io/projected/58dc20f0-d11a-4f2e-becb-b7be97429801-kube-api-access-cq5wm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:27.809592 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.809526 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:27.809592 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:27.809538 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58dc20f0-d11a-4f2e-becb-b7be97429801-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:09:28.495440 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:28.495409 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" Oct 31 04:09:28.495629 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:28.495453 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r" event={"ID":"58dc20f0-d11a-4f2e-becb-b7be97429801","Type":"ContainerDied","Data":"8603a8266a65ec3c302bd5dd80866f23c086e298905e0ca83a377a5f1054c50b"} Oct 31 04:09:28.495629 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:28.495502 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8603a8266a65ec3c302bd5dd80866f23c086e298905e0ca83a377a5f1054c50b" Oct 31 04:09:31.048431 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:31.048388 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v"] Oct 31 04:09:31.098623 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:31.098588 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kww9v"] Oct 31 04:09:32.315951 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:32.315913 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40aa8c13-8d3a-45b5-8df1-c3d6b4546cac" path="/var/lib/kubelet/pods/40aa8c13-8d3a-45b5-8df1-c3d6b4546cac/volumes" Oct 31 04:09:34.204434 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:34.204402 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:35.789446 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:35.789416 4488 scope.go:122] "RemoveContainer" containerID="270aea95d4892444d9f857a74c340e0df1e08f60175f3c35a1ef80d6a4eb7d1b" Oct 31 04:09:36.589461 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:09:36.589424 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:09:39.098320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:39.098286 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:09:40.735276 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:40.735230 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:09:40.735989 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:40.735950 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:09:41.189357 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:41.189300 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:09:41.190191 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:41.189722 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:09:41.193412 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:41.193380 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:09:41.193687 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:41.193656 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:09:43.048871 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:43.048821 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr"] Oct 31 04:09:43.056787 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:43.056760 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jhvrr"] Oct 31 04:09:44.315195 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:44.315105 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81dafaa-66ea-46e1-a00f-9ddba5fb6560" path="/var/lib/kubelet/pods/c81dafaa-66ea-46e1-a00f-9ddba5fb6560/volumes" Oct 31 04:09:49.619276 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:49.619243 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:09:56.227742 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:56.227708 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:56.829988 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:56.829956 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:57.039547 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:09:57.039486 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:01.336688 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:01.336655 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:01.505451 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:01.505416 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:01.661863 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:01.661768 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:04.615365 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:04.615333 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:04.801010 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:04.800980 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:05.182487 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.182446 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc"] Oct 31 04:10:05.183052 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.183025 4488 cpu_manager.go:414] "containerMap: removing container" podUID="58dc20f0-d11a-4f2e-becb-b7be97429801" containerName="revoke-keys" Oct 31 04:10:05.183237 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.183057 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58dc20f0-d11a-4f2e-becb-b7be97429801" containerName="revoke-keys" Oct 31 04:10:05.183237 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.183220 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="58dc20f0-d11a-4f2e-becb-b7be97429801" containerName="revoke-keys" Oct 31 04:10:05.186868 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.186841 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.289162 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.289120 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-20cd9e\"" Oct 31 04:10:05.289162 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.289141 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c59dc76a-dockercfg-qf2j4\"" Oct 31 04:10:05.291734 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.291711 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc"] Oct 31 04:10:05.311570 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.311536 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.311763 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.311688 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgbn\" (UniqueName: \"kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn\") pod \"payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.351402 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.351365 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g"] Oct 31 04:10:05.354838 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.354820 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.413239 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.413203 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgbn\" (UniqueName: \"kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn\") pod \"payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.413495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.413273 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7mr\" (UniqueName: \"kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr\") pod \"payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.413495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.413330 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.413495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.413381 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.413686 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.413535 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g"] Oct 31 04:10:05.414381 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.414353 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.478029 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.477942 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgbn\" (UniqueName: \"kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn\") pod \"payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.504437 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.504400 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:05.514581 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.514545 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7mr\" (UniqueName: \"kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr\") pod \"payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.514713 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.514631 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.515397 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.515364 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.587047 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.586972 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7mr\" (UniqueName: \"kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr\") pod \"payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.668588 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.668551 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:05.882841 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:05.882817 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc"] Oct 31 04:10:05.884837 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:10:05.884794 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a855cbf_d8c0_4638_bde0_528310c51862.slice/crio-49ab74b3212341a4bb9521448989d799ef3bcc3c4d915d34188e4075babd23b7 WatchSource:0}: Error finding container 49ab74b3212341a4bb9521448989d799ef3bcc3c4d915d34188e4075babd23b7: Status 404 returned error can't find the container with id 49ab74b3212341a4bb9521448989d799ef3bcc3c4d915d34188e4075babd23b7 Oct 31 04:10:06.048259 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.048225 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g"] Oct 31 04:10:06.050695 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:10:06.050668 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f97a993_115b_400c_8842_69493d63603b.slice/crio-49f9567dba1203e009b58e5311cfcee72b3ab4e451903dbf0fcf8db5163cd390 WatchSource:0}: Error finding container 49f9567dba1203e009b58e5311cfcee72b3ab4e451903dbf0fcf8db5163cd390: Status 404 returned error can't find the container with id 49f9567dba1203e009b58e5311cfcee72b3ab4e451903dbf0fcf8db5163cd390 Oct 31 04:10:06.593708 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:10:06.593663 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:10:06.709182 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.709141 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" event={"ID":"6f97a993-115b-400c-8842-69493d63603b","Type":"ContainerStarted","Data":"d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616"} Oct 31 04:10:06.709680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.709208 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" event={"ID":"6f97a993-115b-400c-8842-69493d63603b","Type":"ContainerStarted","Data":"49f9567dba1203e009b58e5311cfcee72b3ab4e451903dbf0fcf8db5163cd390"} Oct 31 04:10:06.709680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.709363 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:06.710920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.710897 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" event={"ID":"8a855cbf-d8c0-4638-bde0-528310c51862","Type":"ContainerStarted","Data":"0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5"} Oct 31 04:10:06.711061 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.710940 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" event={"ID":"8a855cbf-d8c0-4638-bde0-528310c51862","Type":"ContainerStarted","Data":"49ab74b3212341a4bb9521448989d799ef3bcc3c4d915d34188e4075babd23b7"} Oct 31 04:10:06.711061 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.711014 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:06.826328 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.826274 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" podStartSLOduration=1.826251839 podStartE2EDuration="1.826251839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:06.824889771 +0000 UTC m=+3034.621608030" watchObservedRunningTime="2026-10-31 04:10:06.826251839 +0000 UTC m=+3034.622970098" Oct 31 04:10:06.945377 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:06.945241 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" podStartSLOduration=1.945220114 podStartE2EDuration="1.945220114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:06.94310126 +0000 UTC m=+3034.739819511" watchObservedRunningTime="2026-10-31 04:10:06.945220114 +0000 UTC m=+3034.741938374" Oct 31 04:10:07.516052 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.516012 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq"] Oct 31 04:10:07.519610 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.519586 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.572207 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.572164 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:10:07.572414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.572223 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:10:07.572414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.572272 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-20cd9e-serving-cert\"" Oct 31 04:10:07.622163 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.622126 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 04:10:07.622402 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.622230 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\"" Oct 31 04:10:07.624840 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.624808 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq"] Oct 31 04:10:07.636928 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.636895 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.637145 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.636969 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlgm\" (UniqueName: \"kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.637145 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.637057 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.637301 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.637169 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.718043 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.718004 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:07.737749 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.737707 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlgm\" (UniqueName: \"kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.737918 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.737764 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.737918 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.737840 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.737918 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.737872 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.738053 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:10:07.738024 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-serving-cert: secret "maas-api-e2e-worker-single-main-20cd9e-serving-cert" not found Oct 31 04:10:07.738135 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:10:07.738129 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls podName:4c215a64-67d9-4ea8-8532-b04af6059bbc nodeName:}" failed. No retries permitted until 2026-10-31 04:10:08.238075297 +0000 UTC m=+3036.034793535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls") pod "maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" (UID: "4c215a64-67d9-4ea8-8532-b04af6059bbc") : secret "maas-api-e2e-worker-single-main-20cd9e-serving-cert" not found Oct 31 04:10:07.738635 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.738573 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.740522 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.740497 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.799680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.799596 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlgm\" (UniqueName: \"kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:07.924149 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:07.924108 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:08.078218 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:08.078173 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:08.243112 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:08.243041 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:08.245693 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:08.245672 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:08.433944 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:08.433845 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:08.813242 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:08.813213 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq"] Oct 31 04:10:08.815137 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:10:08.815047 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c215a64_67d9_4ea8_8532_b04af6059bbc.slice/crio-030495c8b2848d7c070a9c908fada575924ad607a589531b6b9f47311c04d803 WatchSource:0}: Error finding container 030495c8b2848d7c070a9c908fada575924ad607a589531b6b9f47311c04d803: Status 404 returned error can't find the container with id 030495c8b2848d7c070a9c908fada575924ad607a589531b6b9f47311c04d803 Oct 31 04:10:08.816518 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:08.816499 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:10:09.728815 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:09.728766 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" event={"ID":"4c215a64-67d9-4ea8-8532-b04af6059bbc","Type":"ContainerStarted","Data":"c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679"} Oct 31 04:10:09.729014 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:09.728838 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:09.729014 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:09.728861 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" event={"ID":"4c215a64-67d9-4ea8-8532-b04af6059bbc","Type":"ContainerStarted","Data":"030495c8b2848d7c070a9c908fada575924ad607a589531b6b9f47311c04d803"} Oct 31 04:10:09.852785 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:09.852717 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" podStartSLOduration=2.109028908 podStartE2EDuration="2.852698252s" totalImagesPullingTime="743.669344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:09.852423852 +0000 UTC m=+3037.649142113" watchObservedRunningTime="2026-10-31 04:10:09.852698252 +0000 UTC m=+3037.649416527" Oct 31 04:10:12.683608 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:12.683575 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:15.740424 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:15.740394 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:10:15.793103 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:15.793060 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:10:17.628377 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:17.628341 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:17.718522 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:17.718493 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:10:17.718729 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:17.718576 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:10:19.061472 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:19.061428 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb"] Oct 31 04:10:19.111528 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:19.111488 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vxgkb"] Oct 31 04:10:20.315368 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:20.315332 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a2e2f3-0195-400d-af7a-4ed3af59d8a9" path="/var/lib/kubelet/pods/b8a2e2f3-0195-400d-af7a-4ed3af59d8a9/volumes" Oct 31 04:10:22.502065 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:22.502009 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:26.037222 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:26.037184 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:26.652530 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:26.652423 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:26.806334 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:26.806302 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:27.714372 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:27.714334 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:29.047910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:29.047865 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx"] Oct 31 04:10:29.066255 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:29.066218 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xdrrx"] Oct 31 04:10:29.449753 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:29.449721 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:30.315008 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:30.314970 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c8bdb3-b40d-45cc-9165-9251f8a7612c" path="/var/lib/kubelet/pods/78c8bdb3-b40d-45cc-9165-9251f8a7612c/volumes" Oct 31 04:10:32.576822 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:32.576779 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:34.079153 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:34.079115 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:10:34.550462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:34.550378 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:35.811947 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:35.811914 4488 scope.go:122] "RemoveContainer" containerID="5035812e0df313495fcc71e525a0a6f9a285159fb32926a45ff2c7e4aa75b69c" Oct 31 04:10:35.821021 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:35.820998 4488 scope.go:122] "RemoveContainer" containerID="09bbecc7508d9c705ebf012c518498f78406fe103290e463e9e2d3a1ee4678f5" Oct 31 04:10:36.590103 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:10:36.590049 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:10:37.480488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:37.480456 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:42.283358 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:42.283318 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:47.127459 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:47.127430 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:51.947744 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:51.947708 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:56.781068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:10:56.781029 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:00.207605 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:00.207560 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:11:01.586408 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:01.586366 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:05.047443 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:05.047401 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg"] Oct 31 04:11:05.097362 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:05.097320 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v4wlg"] Oct 31 04:11:06.315285 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:06.315246 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cbf0c8-8c50-4510-9df6-97c301949712" path="/var/lib/kubelet/pods/17cbf0c8-8c50-4510-9df6-97c301949712/volumes" Oct 31 04:11:06.483543 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:06.483504 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:06.589531 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:11:06.589493 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:11:11.284464 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:11.284433 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:16.189614 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:16.189533 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:20.997184 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:20.997153 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:25.888476 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:25.888443 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:26.165773 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:26.165688 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:29.411803 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:29.411763 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:11:30.979652 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:30.979611 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:31.259615 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:31.259523 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:33.047888 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:33.047843 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw"] Oct 31 04:11:33.097984 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:33.097948 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-82wrw"] Oct 31 04:11:34.315453 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:34.315414 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcc2445-b145-45a1-91df-bb1c768b86c6" path="/var/lib/kubelet/pods/9bcc2445-b145-45a1-91df-bb1c768b86c6/volumes" Oct 31 04:11:34.899913 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:34.899840 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:35.846967 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:35.846925 4488 scope.go:122] "RemoveContainer" containerID="4e7a893b46b0a26dfe1d8f4e895040810402b93bb69e46f677931e51d878790a" Oct 31 04:11:35.856162 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:35.856138 4488 scope.go:122] "RemoveContainer" containerID="a9f6aa32800bf2fdab7ecb1fc43e6fc1c2717e8eb83f929a6cbf1f21c640037e" Oct 31 04:11:35.864794 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:35.864768 4488 scope.go:122] "RemoveContainer" containerID="0021aa18729897d9734c3fcdfd6d63175f552935d9c82b2c2d1246c38d6a38dc" Oct 31 04:11:35.929400 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:35.929360 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:36.589596 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:11:36.589547 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:11:38.240493 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:38.240454 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc"] Oct 31 04:11:38.348932 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:38.348900 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g"] Oct 31 04:11:38.617905 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:38.617868 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:38.774317 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:38.774285 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:40.688333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:40.688300 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:43.240996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:43.240932 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" containerName="payload-processing" containerID="cri-o://0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5" gracePeriod=25 Oct 31 04:11:43.349591 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:43.349547 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" podUID="6f97a993-115b-400c-8842-69493d63603b" containerName="payload-pre-processing" containerID="cri-o://d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616" gracePeriod=25 Oct 31 04:11:44.810318 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:44.810285 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq"] Oct 31 04:11:44.810755 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:44.810527 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" podUID="4c215a64-67d9-4ea8-8532-b04af6059bbc" containerName="maas-api" containerID="cri-o://c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679" gracePeriod=30 Oct 31 04:11:44.957812 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:44.957778 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:45.735210 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:45.735150 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" podUID="4c215a64-67d9-4ea8-8532-b04af6059bbc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.213:8443/health\": dial tcp 10.133.0.213:8443: connect: connection refused" Oct 31 04:11:46.062278 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.062252 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:11:46.202560 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.202530 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca\") pod \"4c215a64-67d9-4ea8-8532-b04af6059bbc\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " Oct 31 04:11:46.202748 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.202675 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs\") pod \"4c215a64-67d9-4ea8-8532-b04af6059bbc\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " Oct 31 04:11:46.202748 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.202717 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\") pod \"4c215a64-67d9-4ea8-8532-b04af6059bbc\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " Oct 31 04:11:46.202748 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.202738 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm\" (UniqueName: \"kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm\") pod \"4c215a64-67d9-4ea8-8532-b04af6059bbc\" (UID: \"4c215a64-67d9-4ea8-8532-b04af6059bbc\") " Oct 31 04:11:46.203033 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.202999 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca" pod "4c215a64-67d9-4ea8-8532-b04af6059bbc" (UID: "4c215a64-67d9-4ea8-8532-b04af6059bbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:46.205112 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.205050 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls" pod "4c215a64-67d9-4ea8-8532-b04af6059bbc" (UID: "4c215a64-67d9-4ea8-8532-b04af6059bbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:46.205112 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.205066 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs" pod "4c215a64-67d9-4ea8-8532-b04af6059bbc" (UID: "4c215a64-67d9-4ea8-8532-b04af6059bbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:46.205284 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.205132 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm" pod "4c215a64-67d9-4ea8-8532-b04af6059bbc" (UID: "4c215a64-67d9-4ea8-8532-b04af6059bbc"). InnerVolumeSpecName "kube-api-access-djlgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:46.230960 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.230929 4488 generic.go:356] "Generic (PLEG): container finished" podID="4c215a64-67d9-4ea8-8532-b04af6059bbc" containerID="c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679" exitCode=0 Oct 31 04:11:46.231163 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.231021 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" event={"ID":"4c215a64-67d9-4ea8-8532-b04af6059bbc","Type":"ContainerDied","Data":"c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679"} Oct 31 04:11:46.231163 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.231036 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" Oct 31 04:11:46.231163 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.231062 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq" event={"ID":"4c215a64-67d9-4ea8-8532-b04af6059bbc","Type":"ContainerDied","Data":"030495c8b2848d7c070a9c908fada575924ad607a589531b6b9f47311c04d803"} Oct 31 04:11:46.231163 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.231112 4488 scope.go:122] "RemoveContainer" containerID="c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679" Oct 31 04:11:46.241238 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.241214 4488 scope.go:122] "RemoveContainer" containerID="c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679" Oct 31 04:11:46.241589 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:11:46.241567 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679\": container with ID starting with c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679 not found: ID does not exist" containerID="c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679" Oct 31 04:11:46.241675 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.241599 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679"} err="failed to get container status \"c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679\": rpc error: code = NotFound desc = could not find container \"c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679\": container with ID starting with c48fe9ed82dcb9a474a319176022cae22a4aa8211224ab55f8ce9dad92247679 not found: ID does not exist" Oct 31 04:11:46.304333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.304236 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.304333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.304269 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c215a64-67d9-4ea8-8532-b04af6059bbc-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.304333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.304282 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djlgm\" (UniqueName: \"kubernetes.io/projected/4c215a64-67d9-4ea8-8532-b04af6059bbc-kube-api-access-djlgm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.304333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.304292 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c215a64-67d9-4ea8-8532-b04af6059bbc-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:46.396275 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.396221 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq"] Oct 31 04:11:46.398367 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.398325 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-20cd9e-6779cf6cfc-z54cq"] Oct 31 04:11:46.419371 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.419342 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:46.705363 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:46.705333 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:47.558825 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:47.558793 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:47.717926 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:47.717630 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" podUID="6f97a993-115b-400c-8842-69493d63603b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:11:47.717926 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:47.717912 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:11:48.315149 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:48.315117 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c215a64-67d9-4ea8-8532-b04af6059bbc" path="/var/lib/kubelet/pods/4c215a64-67d9-4ea8-8532-b04af6059bbc/volumes" Oct 31 04:11:49.565472 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:49.565426 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:11:53.414745 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:53.414705 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:53.674414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:53.674326 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:53.830323 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:53.830284 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:55.381741 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.381706 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r"] Oct 31 04:11:55.431557 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.431519 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rf57r"] Oct 31 04:11:55.509608 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.509572 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2"] Oct 31 04:11:55.509956 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.509944 4488 cpu_manager.go:414] "containerMap: removing container" podUID="4c215a64-67d9-4ea8-8532-b04af6059bbc" containerName="maas-api" Oct 31 04:11:55.510009 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.509957 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c215a64-67d9-4ea8-8532-b04af6059bbc" containerName="maas-api" Oct 31 04:11:55.510056 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.510049 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c215a64-67d9-4ea8-8532-b04af6059bbc" containerName="maas-api" Oct 31 04:11:55.513130 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.513106 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.566125 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.566060 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:11:55.566359 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.566267 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:11:55.566359 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.566282 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:11:55.566359 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.566298 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:11:55.577736 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.577703 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2"] Oct 31 04:11:55.694303 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.694199 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2lw\" (UniqueName: \"kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.694303 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.694243 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.694524 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.694395 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.795747 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.795703 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2lw\" (UniqueName: \"kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.795747 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.795753 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.796023 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.795836 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.796634 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.796609 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.798512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.798493 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:55.857891 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:55.857858 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2lw\" (UniqueName: \"kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw\") pod \"maas-api-revoke-sub-3d4156f1-2n4r2\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:56.126730 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:56.126678 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:56.315587 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:56.315549 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58dc20f0-d11a-4f2e-becb-b7be97429801" path="/var/lib/kubelet/pods/58dc20f0-d11a-4f2e-becb-b7be97429801/volumes" Oct 31 04:11:56.502043 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:56.501811 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2"] Oct 31 04:11:56.505299 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:11:56.505268 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ace00f_8f79_4c36_b460_1987ccabc1fa.slice/crio-948c96ee33b4471168529c14f52dd56df11ec7b2cd99795d8ae744ea0d182b67 WatchSource:0}: Error finding container 948c96ee33b4471168529c14f52dd56df11ec7b2cd99795d8ae744ea0d182b67: Status 404 returned error can't find the container with id 948c96ee33b4471168529c14f52dd56df11ec7b2cd99795d8ae744ea0d182b67 Oct 31 04:11:57.289402 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:57.289362 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" event={"ID":"07ace00f-8f79-4c36-b460-1987ccabc1fa","Type":"ContainerStarted","Data":"a4445699e3e4631f789882cfe26d8cd7948d3f88cc839259892751a539af63a5"} Oct 31 04:11:57.289600 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:57.289429 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" event={"ID":"07ace00f-8f79-4c36-b460-1987ccabc1fa","Type":"ContainerStarted","Data":"948c96ee33b4471168529c14f52dd56df11ec7b2cd99795d8ae744ea0d182b67"} Oct 31 04:11:57.406557 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:57.406505 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" podStartSLOduration=2.406487177 podStartE2EDuration="2.406487177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:57.405880902 +0000 UTC m=+3145.202599160" watchObservedRunningTime="2026-10-31 04:11:57.406487177 +0000 UTC m=+3145.203205436" Oct 31 04:11:57.717179 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:57.717137 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" podUID="6f97a993-115b-400c-8842-69493d63603b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:11:57.717601 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:57.717141 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:11:58.295497 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:58.295461 4488 generic.go:356] "Generic (PLEG): container finished" podID="07ace00f-8f79-4c36-b460-1987ccabc1fa" containerID="a4445699e3e4631f789882cfe26d8cd7948d3f88cc839259892751a539af63a5" exitCode=0 Oct 31 04:11:58.295696 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:58.295536 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" event={"ID":"07ace00f-8f79-4c36-b460-1987ccabc1fa","Type":"ContainerDied","Data":"a4445699e3e4631f789882cfe26d8cd7948d3f88cc839259892751a539af63a5"} Oct 31 04:11:59.438106 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.438071 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:11:59.531900 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.531856 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca\") pod \"07ace00f-8f79-4c36-b460-1987ccabc1fa\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " Oct 31 04:11:59.532124 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.531924 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw\" (UniqueName: \"kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw\") pod \"07ace00f-8f79-4c36-b460-1987ccabc1fa\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " Oct 31 04:11:59.532124 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.531979 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca\") pod \"07ace00f-8f79-4c36-b460-1987ccabc1fa\" (UID: \"07ace00f-8f79-4c36-b460-1987ccabc1fa\") " Oct 31 04:11:59.532438 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.532410 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca" pod "07ace00f-8f79-4c36-b460-1987ccabc1fa" (UID: "07ace00f-8f79-4c36-b460-1987ccabc1fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:59.534241 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.534209 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca" pod "07ace00f-8f79-4c36-b460-1987ccabc1fa" (UID: "07ace00f-8f79-4c36-b460-1987ccabc1fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:59.534346 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.534247 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw" pod "07ace00f-8f79-4c36-b460-1987ccabc1fa" (UID: "07ace00f-8f79-4c36-b460-1987ccabc1fa"). InnerVolumeSpecName "kube-api-access-xf2lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:59.632662 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.632625 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.632662 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.632656 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2lw\" (UniqueName: \"kubernetes.io/projected/07ace00f-8f79-4c36-b460-1987ccabc1fa-kube-api-access-xf2lw\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:11:59.632662 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:11:59.632666 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace00f-8f79-4c36-b460-1987ccabc1fa-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:00.306661 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:00.306629 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" Oct 31 04:12:00.306661 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:00.306651 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2" event={"ID":"07ace00f-8f79-4c36-b460-1987ccabc1fa","Type":"ContainerDied","Data":"948c96ee33b4471168529c14f52dd56df11ec7b2cd99795d8ae744ea0d182b67"} Oct 31 04:12:00.306915 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:00.306693 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948c96ee33b4471168529c14f52dd56df11ec7b2cd99795d8ae744ea0d182b67" Oct 31 04:12:03.048526 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.048488 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln"] Oct 31 04:12:03.098631 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.098592 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9lmln"] Oct 31 04:12:03.507400 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.507367 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:12:03.560714 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.560672 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume\") pod \"8a855cbf-d8c0-4638-bde0-528310c51862\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " Oct 31 04:12:03.560932 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.560789 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn\" (UniqueName: \"kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn\") pod \"8a855cbf-d8c0-4638-bde0-528310c51862\" (UID: \"8a855cbf-d8c0-4638-bde0-528310c51862\") " Oct 31 04:12:03.561060 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.561037 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume" pod "8a855cbf-d8c0-4638-bde0-528310c51862" (UID: "8a855cbf-d8c0-4638-bde0-528310c51862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:03.563251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.563219 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn" pod "8a855cbf-d8c0-4638-bde0-528310c51862" (UID: "8a855cbf-d8c0-4638-bde0-528310c51862"). InnerVolumeSpecName "kube-api-access-djgbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:03.609576 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.609501 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:12:03.661994 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.661962 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr\" (UniqueName: \"kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr\") pod \"6f97a993-115b-400c-8842-69493d63603b\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " Oct 31 04:12:03.661994 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.662009 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume\") pod \"6f97a993-115b-400c-8842-69493d63603b\" (UID: \"6f97a993-115b-400c-8842-69493d63603b\") " Oct 31 04:12:03.662293 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.662201 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgbn\" (UniqueName: \"kubernetes.io/projected/8a855cbf-d8c0-4638-bde0-528310c51862-kube-api-access-djgbn\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:03.662293 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.662213 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a855cbf-d8c0-4638-bde0-528310c51862-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:03.662497 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.662472 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume" pod "6f97a993-115b-400c-8842-69493d63603b" (UID: "6f97a993-115b-400c-8842-69493d63603b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:03.664386 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.664359 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr" pod "6f97a993-115b-400c-8842-69493d63603b" (UID: "6f97a993-115b-400c-8842-69493d63603b"). InnerVolumeSpecName "kube-api-access-pc7mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:03.763117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.763059 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc7mr\" (UniqueName: \"kubernetes.io/projected/6f97a993-115b-400c-8842-69493d63603b-kube-api-access-pc7mr\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:03.763117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:03.763106 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f97a993-115b-400c-8842-69493d63603b-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:12:04.315585 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.315548 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2ce102-b329-425c-96a6-b9dcedd8805e" path="/var/lib/kubelet/pods/ec2ce102-b329-425c-96a6-b9dcedd8805e/volumes" Oct 31 04:12:04.328952 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.328925 4488 generic.go:356] "Generic (PLEG): container finished" podID="8a855cbf-d8c0-4638-bde0-528310c51862" containerID="0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5" exitCode=0 Oct 31 04:12:04.329127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.328965 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" Oct 31 04:12:04.329127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.329000 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" event={"ID":"8a855cbf-d8c0-4638-bde0-528310c51862","Type":"ContainerDied","Data":"0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5"} Oct 31 04:12:04.329127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.329056 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc" event={"ID":"8a855cbf-d8c0-4638-bde0-528310c51862","Type":"ContainerDied","Data":"49ab74b3212341a4bb9521448989d799ef3bcc3c4d915d34188e4075babd23b7"} Oct 31 04:12:04.329127 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.329081 4488 scope.go:122] "RemoveContainer" containerID="0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5" Oct 31 04:12:04.330558 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.330531 4488 generic.go:356] "Generic (PLEG): container finished" podID="6f97a993-115b-400c-8842-69493d63603b" containerID="d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616" exitCode=0 Oct 31 04:12:04.330849 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.330578 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" Oct 31 04:12:04.330849 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.330605 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" event={"ID":"6f97a993-115b-400c-8842-69493d63603b","Type":"ContainerDied","Data":"d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616"} Oct 31 04:12:04.330849 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.330659 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g" event={"ID":"6f97a993-115b-400c-8842-69493d63603b","Type":"ContainerDied","Data":"49f9567dba1203e009b58e5311cfcee72b3ab4e451903dbf0fcf8db5163cd390"} Oct 31 04:12:04.338367 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.338345 4488 scope.go:122] "RemoveContainer" containerID="0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5" Oct 31 04:12:04.338715 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:12:04.338691 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5\": container with ID starting with 0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5 not found: ID does not exist" containerID="0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5" Oct 31 04:12:04.338785 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.338727 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5"} err="failed to get container status \"0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5\": rpc error: code = NotFound desc = could not find container \"0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5\": container with ID starting with 0ff98a29c49227d93664280431c2bd97e9242e8a8af60c707589ef54e01d9cb5 not found: ID does not exist" Oct 31 04:12:04.338785 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.338748 4488 scope.go:122] "RemoveContainer" containerID="d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616" Oct 31 04:12:04.347877 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.347857 4488 scope.go:122] "RemoveContainer" containerID="d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616" Oct 31 04:12:04.348260 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:12:04.348240 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616\": container with ID starting with d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616 not found: ID does not exist" containerID="d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616" Oct 31 04:12:04.348315 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.348270 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616"} err="failed to get container status \"d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616\": rpc error: code = NotFound desc = could not find container \"d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616\": container with ID starting with d2b4f828483b1890985c031dd6cb510a1677f6b36ac1b5a1ecf3f78616d2a616 not found: ID does not exist" Oct 31 04:12:04.494158 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.494121 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc"] Oct 31 04:12:04.544208 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.544115 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-20cd9e-685bcfc848wwbc"] Oct 31 04:12:04.608082 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.608044 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g"] Oct 31 04:12:04.610409 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:04.610377 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-20cd9e-58cd9jks5g"] Oct 31 04:12:06.316505 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:06.316468 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f97a993-115b-400c-8842-69493d63603b" path="/var/lib/kubelet/pods/6f97a993-115b-400c-8842-69493d63603b/volumes" Oct 31 04:12:06.316977 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:06.316959 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" path="/var/lib/kubelet/pods/8a855cbf-d8c0-4638-bde0-528310c51862/volumes" Oct 31 04:12:06.589315 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:12:06.589276 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:12:07.662120 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:07.662065 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:08.928703 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:08.928661 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:12.387937 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:12.387878 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:12:14.570797 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:14.570759 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:23.091844 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:23.091805 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:12:27.046348 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:27.046309 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg"] Oct 31 04:12:27.096187 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:27.096157 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sf5wg"] Oct 31 04:12:28.314924 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:28.314886 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b926d8d0-47f9-4f8f-a7bc-4ca0f8657937" path="/var/lib/kubelet/pods/b926d8d0-47f9-4f8f-a7bc-4ca0f8657937/volumes" Oct 31 04:12:35.251662 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:35.251630 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:35.897926 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:35.897892 4488 scope.go:122] "RemoveContainer" containerID="89cb611773f9b251237f3ccca74dbf55adaa05fcd78002aef7dceb2283b5594b" Oct 31 04:12:35.906871 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:12:35.906848 4488 scope.go:122] "RemoveContainer" containerID="67254708ba4d71255fd0e9216d6b0dd14cdd7271fa80a18e8c08e68178548cc7" Oct 31 04:12:36.589178 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:12:36.589141 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:13:06.589225 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:13:06.589183 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:13:35.601467 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:13:35.601417 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:35.942139 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:13:35.942004 4488 scope.go:122] "RemoveContainer" containerID="95ca8a9138248d98fa334d22960ce213d449ef142cdb73a76ae564289a608ac1" Oct 31 04:13:35.950470 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:13:35.950447 4488 scope.go:122] "RemoveContainer" containerID="7ee5e15d6abf4182c7e966fda3b22f250ad49f70f5ea7fd36625cccce61e63de" Oct 31 04:13:35.958287 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:13:35.958266 4488 scope.go:122] "RemoveContainer" containerID="ce8d2c849d896a97863ef04b3c32c4fb52dd29739d2f8a6851b0a30b44f3a5bf" Oct 31 04:13:36.589697 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:13:36.589661 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:13:59.488384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:13:59.488346 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:14:00.048552 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:00.048511 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh"] Oct 31 04:14:00.098611 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:00.098566 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5f4wh"] Oct 31 04:14:00.319266 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:00.319166 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5339dd-2599-42ce-9179-2af9a5264f1d" path="/var/lib/kubelet/pods/ec5339dd-2599-42ce-9179-2af9a5264f1d/volumes" Oct 31 04:14:04.113568 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.113526 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4"] Oct 31 04:14:04.114147 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114127 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6f97a993-115b-400c-8842-69493d63603b" containerName="payload-pre-processing" Oct 31 04:14:04.114251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114148 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f97a993-115b-400c-8842-69493d63603b" containerName="payload-pre-processing" Oct 31 04:14:04.114251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114181 4488 cpu_manager.go:414] "containerMap: removing container" podUID="07ace00f-8f79-4c36-b460-1987ccabc1fa" containerName="revoke-keys" Oct 31 04:14:04.114251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114189 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ace00f-8f79-4c36-b460-1987ccabc1fa" containerName="revoke-keys" Oct 31 04:14:04.114251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114213 4488 cpu_manager.go:414] "containerMap: removing container" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" containerName="payload-processing" Oct 31 04:14:04.114251 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114221 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" containerName="payload-processing" Oct 31 04:14:04.114516 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114320 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f97a993-115b-400c-8842-69493d63603b" containerName="payload-pre-processing" Oct 31 04:14:04.114516 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114335 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ace00f-8f79-4c36-b460-1987ccabc1fa" containerName="revoke-keys" Oct 31 04:14:04.114516 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.114346 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a855cbf-d8c0-4638-bde0-528310c51862" containerName="payload-processing" Oct 31 04:14:04.119133 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.119110 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.214604 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.214565 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcrd\" (UniqueName: \"kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.214797 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.214628 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.214797 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.214650 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.222755 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.222723 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:14:04.222755 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.222747 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:14:04.223052 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.222776 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:14:04.223052 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.222794 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:14:04.226910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.226881 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4"] Oct 31 04:14:04.316332 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.316295 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcrd\" (UniqueName: \"kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.316566 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.316368 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.316566 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.316399 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.317071 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.317043 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.318963 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.318943 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.377995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.377919 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcrd\" (UniqueName: \"kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd\") pod \"maas-api-revoke-sub-d576e68b-kb2k4\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.432835 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.432802 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:04.808884 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.808855 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4"] Oct 31 04:14:04.811425 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:14:04.811393 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7842fcf2_e087_4208_8ed6_37334c98e188.slice/crio-4b943a1dd80a19584413760e596fb0b2eb24f5dbba0532a696b89d13c906fe97 WatchSource:0}: Error finding container 4b943a1dd80a19584413760e596fb0b2eb24f5dbba0532a696b89d13c906fe97: Status 404 returned error can't find the container with id 4b943a1dd80a19584413760e596fb0b2eb24f5dbba0532a696b89d13c906fe97 Oct 31 04:14:04.974537 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:04.974503 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" event={"ID":"7842fcf2-e087-4208-8ed6-37334c98e188","Type":"ContainerStarted","Data":"4b943a1dd80a19584413760e596fb0b2eb24f5dbba0532a696b89d13c906fe97"} Oct 31 04:14:05.981229 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:05.981124 4488 generic.go:356] "Generic (PLEG): container finished" podID="7842fcf2-e087-4208-8ed6-37334c98e188" containerID="ea9932716bacc6b3471a8d3d0ccf874268232d3ff7f9993cda3992d2c00273d4" exitCode=0 Oct 31 04:14:05.981229 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:05.981215 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" event={"ID":"7842fcf2-e087-4208-8ed6-37334c98e188","Type":"ContainerDied","Data":"ea9932716bacc6b3471a8d3d0ccf874268232d3ff7f9993cda3992d2c00273d4"} Oct 31 04:14:06.588819 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:14:06.588785 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:14:07.120360 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.120333 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:07.139520 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.139479 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca\") pod \"7842fcf2-e087-4208-8ed6-37334c98e188\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " Oct 31 04:14:07.139716 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.139590 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca\") pod \"7842fcf2-e087-4208-8ed6-37334c98e188\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " Oct 31 04:14:07.139716 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.139674 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd\" (UniqueName: \"kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd\") pod \"7842fcf2-e087-4208-8ed6-37334c98e188\" (UID: \"7842fcf2-e087-4208-8ed6-37334c98e188\") " Oct 31 04:14:07.139973 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.139949 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca" pod "7842fcf2-e087-4208-8ed6-37334c98e188" (UID: "7842fcf2-e087-4208-8ed6-37334c98e188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:07.141715 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.141687 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca" pod "7842fcf2-e087-4208-8ed6-37334c98e188" (UID: "7842fcf2-e087-4208-8ed6-37334c98e188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:07.141835 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.141789 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd" pod "7842fcf2-e087-4208-8ed6-37334c98e188" (UID: "7842fcf2-e087-4208-8ed6-37334c98e188"). InnerVolumeSpecName "kube-api-access-5jcrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:07.240495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.240416 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jcrd\" (UniqueName: \"kubernetes.io/projected/7842fcf2-e087-4208-8ed6-37334c98e188-kube-api-access-5jcrd\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:14:07.240495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.240448 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:14:07.240495 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.240458 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842fcf2-e087-4208-8ed6-37334c98e188-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:14:07.991726 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.991695 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" Oct 31 04:14:07.991919 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.991694 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4" event={"ID":"7842fcf2-e087-4208-8ed6-37334c98e188","Type":"ContainerDied","Data":"4b943a1dd80a19584413760e596fb0b2eb24f5dbba0532a696b89d13c906fe97"} Oct 31 04:14:07.991919 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:07.991820 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b943a1dd80a19584413760e596fb0b2eb24f5dbba0532a696b89d13c906fe97" Oct 31 04:14:11.047957 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:11.047902 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl"] Oct 31 04:14:11.097660 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:11.097622 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-sv2cl"] Oct 31 04:14:12.315497 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:12.315458 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ad9320-e3d1-45fe-87f0-a63040fb6fa7" path="/var/lib/kubelet/pods/43ad9320-e3d1-45fe-87f0-a63040fb6fa7/volumes" Oct 31 04:14:16.963628 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:16.963535 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:14:26.562244 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:26.562205 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:14:33.271343 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.271301 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg"] Oct 31 04:14:33.272039 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.271692 4488 cpu_manager.go:414] "containerMap: removing container" podUID="7842fcf2-e087-4208-8ed6-37334c98e188" containerName="revoke-keys" Oct 31 04:14:33.272039 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.271703 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7842fcf2-e087-4208-8ed6-37334c98e188" containerName="revoke-keys" Oct 31 04:14:33.272039 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.271782 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="7842fcf2-e087-4208-8ed6-37334c98e188" containerName="revoke-keys" Oct 31 04:14:33.276228 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.276196 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.330906 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.330847 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:14:33.380591 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.380547 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:14:33.380796 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.380618 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:14:33.380796 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.380768 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:14:33.382454 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.382430 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg"] Oct 31 04:14:33.436973 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.436934 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.437194 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.436998 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.437194 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.437115 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r5hn\" (UniqueName: \"kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.538304 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.538215 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.538304 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.538255 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r5hn\" (UniqueName: \"kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.538515 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.538352 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.539021 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.538989 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.540780 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.540753 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.599984 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.599950 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r5hn\" (UniqueName: \"kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn\") pod \"maas-api-revoke-sub-d2bc93bc-pwxlg\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:33.890271 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:33.890229 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:34.267437 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:34.267410 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg"] Oct 31 04:14:34.270182 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:14:34.270157 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7a6179_3ad1_43ce_be1c_6427da8ddad7.slice/crio-6b59769a38afbf32adbd6366fd1f8d9c79ee31ffc9263de9bab339f3e960a8ce WatchSource:0}: Error finding container 6b59769a38afbf32adbd6366fd1f8d9c79ee31ffc9263de9bab339f3e960a8ce: Status 404 returned error can't find the container with id 6b59769a38afbf32adbd6366fd1f8d9c79ee31ffc9263de9bab339f3e960a8ce Oct 31 04:14:35.153115 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:35.153055 4488 generic.go:356] "Generic (PLEG): container finished" podID="ed7a6179-3ad1-43ce-be1c-6427da8ddad7" containerID="25ad6b666d44b7ca3f7d348d372295f45434e5738b6210a24d1395b45a258670" exitCode=0 Oct 31 04:14:35.153583 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:35.153147 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" event={"ID":"ed7a6179-3ad1-43ce-be1c-6427da8ddad7","Type":"ContainerDied","Data":"25ad6b666d44b7ca3f7d348d372295f45434e5738b6210a24d1395b45a258670"} Oct 31 04:14:35.153583 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:35.153215 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" event={"ID":"ed7a6179-3ad1-43ce-be1c-6427da8ddad7","Type":"ContainerStarted","Data":"6b59769a38afbf32adbd6366fd1f8d9c79ee31ffc9263de9bab339f3e960a8ce"} Oct 31 04:14:35.983872 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:35.983840 4488 scope.go:122] "RemoveContainer" containerID="7ac892530b97ac8863663474906bd87d4867b4561c9c1db425a1f131a3e81bf3" Oct 31 04:14:36.307901 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.307867 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:36.466203 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.466169 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca\") pod \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " Oct 31 04:14:36.466394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.466209 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca\") pod \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " Oct 31 04:14:36.466394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.466281 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn\" (UniqueName: \"kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn\") pod \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\" (UID: \"ed7a6179-3ad1-43ce-be1c-6427da8ddad7\") " Oct 31 04:14:36.466621 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.466598 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca" pod "ed7a6179-3ad1-43ce-be1c-6427da8ddad7" (UID: "ed7a6179-3ad1-43ce-be1c-6427da8ddad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:36.466734 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.466715 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:14:36.468622 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.468595 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca" pod "ed7a6179-3ad1-43ce-be1c-6427da8ddad7" (UID: "ed7a6179-3ad1-43ce-be1c-6427da8ddad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:36.468622 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.468609 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn" pod "ed7a6179-3ad1-43ce-be1c-6427da8ddad7" (UID: "ed7a6179-3ad1-43ce-be1c-6427da8ddad7"). InnerVolumeSpecName "kube-api-access-2r5hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:36.568073 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.567980 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:14:36.568073 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:36.568016 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r5hn\" (UniqueName: \"kubernetes.io/projected/ed7a6179-3ad1-43ce-be1c-6427da8ddad7-kube-api-access-2r5hn\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:14:36.590083 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:14:36.590044 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:14:37.104384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:37.104345 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:37.165295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:37.165260 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" Oct 31 04:14:37.165295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:37.165275 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg" event={"ID":"ed7a6179-3ad1-43ce-be1c-6427da8ddad7","Type":"ContainerDied","Data":"6b59769a38afbf32adbd6366fd1f8d9c79ee31ffc9263de9bab339f3e960a8ce"} Oct 31 04:14:37.165565 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:37.165337 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b59769a38afbf32adbd6366fd1f8d9c79ee31ffc9263de9bab339f3e960a8ce" Oct 31 04:14:45.647907 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:45.647868 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:46.111694 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:46.111657 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:14:46.369680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:46.369590 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:46.582815 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:46.582782 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:50.630156 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:50.630114 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:50.808020 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:50.807983 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:50.969682 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:50.969593 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:51.141825 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:51.141790 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:53.953003 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:53.952972 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:54.127724 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.127686 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:54.474485 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.474439 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5"] Oct 31 04:14:54.474866 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.474850 4488 cpu_manager.go:414] "containerMap: removing container" podUID="ed7a6179-3ad1-43ce-be1c-6427da8ddad7" containerName="revoke-keys" Oct 31 04:14:54.474866 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.474866 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7a6179-3ad1-43ce-be1c-6427da8ddad7" containerName="revoke-keys" Oct 31 04:14:54.475006 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.474959 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7a6179-3ad1-43ce-be1c-6427da8ddad7" containerName="revoke-keys" Oct 31 04:14:54.485221 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.485178 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.537978 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.537937 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5"] Oct 31 04:14:54.545714 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.545677 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqs5d\" (UniqueName: \"kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d\") pod \"payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.545916 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.545829 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.585776 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.585739 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-09c573\"" Oct 31 04:14:54.586004 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.585831 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-48c73e64-dockercfg-6qg8h\"" Oct 31 04:14:54.646572 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.646531 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqs5d\" (UniqueName: \"kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d\") pod \"payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.646779 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.646595 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.647325 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.647305 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.664003 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.663966 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l"] Oct 31 04:14:54.667774 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.667751 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.708605 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.708565 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqs5d\" (UniqueName: \"kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d\") pod \"payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.726368 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.726274 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l"] Oct 31 04:14:54.748104 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.748051 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h227v\" (UniqueName: \"kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v\") pod \"payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.748334 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.748122 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.805282 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.805244 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:54.848982 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.848947 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h227v\" (UniqueName: \"kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v\") pod \"payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.849185 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.848997 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.849708 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.849688 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.910729 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.910695 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h227v\" (UniqueName: \"kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v\") pod \"payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:54.981046 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:54.980949 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:55.180813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:55.180788 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5"] Oct 31 04:14:55.183046 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:14:55.183016 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa50cc2a_ce0c_40d6_b0ae_75a8f2e3ec47.slice/crio-3adb640e999db84389eb4ec58f3c10a890bcf1281ab48d07b72b2ae81cae80bd WatchSource:0}: Error finding container 3adb640e999db84389eb4ec58f3c10a890bcf1281ab48d07b72b2ae81cae80bd: Status 404 returned error can't find the container with id 3adb640e999db84389eb4ec58f3c10a890bcf1281ab48d07b72b2ae81cae80bd Oct 31 04:14:55.258304 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:55.258271 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" event={"ID":"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47","Type":"ContainerStarted","Data":"f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833"} Oct 31 04:14:55.258454 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:55.258334 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" event={"ID":"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47","Type":"ContainerStarted","Data":"3adb640e999db84389eb4ec58f3c10a890bcf1281ab48d07b72b2ae81cae80bd"} Oct 31 04:14:55.258512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:55.258496 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:14:55.357380 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:55.357357 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l"] Oct 31 04:14:55.359427 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:14:55.359396 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd22191_42e5_47f7_bd46_61b046d458b4.slice/crio-7d5991dfde8f4cc6fdf0f5b5ad6697fcf08778fa5e4b4b3a2730ab01382f90cf WatchSource:0}: Error finding container 7d5991dfde8f4cc6fdf0f5b5ad6697fcf08778fa5e4b4b3a2730ab01382f90cf: Status 404 returned error can't find the container with id 7d5991dfde8f4cc6fdf0f5b5ad6697fcf08778fa5e4b4b3a2730ab01382f90cf Oct 31 04:14:55.375148 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:55.375079 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" podStartSLOduration=1.375066117 podStartE2EDuration="1.375066117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:55.374453783 +0000 UTC m=+3323.171172054" watchObservedRunningTime="2026-10-31 04:14:55.375066117 +0000 UTC m=+3323.171784375" Oct 31 04:14:56.264533 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.264495 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" event={"ID":"abd22191-42e5-47f7-bd46-61b046d458b4","Type":"ContainerStarted","Data":"f7b2672a1cb87540aca448123082f6a4e972f6598e533da0130f5a865f857937"} Oct 31 04:14:56.264967 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.264554 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" event={"ID":"abd22191-42e5-47f7-bd46-61b046d458b4","Type":"ContainerStarted","Data":"7d5991dfde8f4cc6fdf0f5b5ad6697fcf08778fa5e4b4b3a2730ab01382f90cf"} Oct 31 04:14:56.264967 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.264758 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:14:56.387485 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.387426 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" podStartSLOduration=2.387410911 podStartE2EDuration="2.387410911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:56.385470634 +0000 UTC m=+3324.182188893" watchObservedRunningTime="2026-10-31 04:14:56.387410911 +0000 UTC m=+3324.184129169" Oct 31 04:14:56.819563 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.819528 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7"] Oct 31 04:14:56.823256 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.823233 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.868379 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.868341 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.868578 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.868421 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l487\" (UniqueName: \"kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.868578 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.868540 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.868655 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.868593 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.876169 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.876138 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-09c573-serving-cert\"" Oct 31 04:14:56.876361 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.876201 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:14:56.876435 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.876363 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 04:14:56.876435 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.876410 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:14:56.926073 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.926039 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\"" Oct 31 04:14:56.928105 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.928058 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7"] Oct 31 04:14:56.969732 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.969694 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l487\" (UniqueName: \"kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.969945 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.969752 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.969945 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.969876 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.969945 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.969930 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.970215 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:14:56.970083 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-serving-cert: secret "maas-api-e2e-worker-single-main-09c573-serving-cert" not found Oct 31 04:14:56.970215 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:14:56.970194 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls podName:80b08c34-5174-4020-be20-124fad799d14 nodeName:}" failed. No retries permitted until 2026-10-31 04:14:57.470169369 +0000 UTC m=+3325.266887620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls") pod "maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" (UID: "80b08c34-5174-4020-be20-124fad799d14") : secret "maas-api-e2e-worker-single-main-09c573-serving-cert" not found Oct 31 04:14:56.970626 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.970603 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:56.972773 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:56.972746 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:57.032004 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.031976 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l487\" (UniqueName: \"kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:57.079444 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.079405 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:57.234180 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.234145 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:57.392228 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.392137 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:57.475509 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.475473 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:57.478016 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.477992 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:57.736867 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:57.736773 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:58.115121 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:58.114849 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7"] Oct 31 04:14:58.117431 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:14:58.117398 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b08c34_5174_4020_be20_124fad799d14.slice/crio-1b9367b65613af007307f21b484ca5f764da67728fe0c094e9ac12b82f6a2d25 WatchSource:0}: Error finding container 1b9367b65613af007307f21b484ca5f764da67728fe0c094e9ac12b82f6a2d25: Status 404 returned error can't find the container with id 1b9367b65613af007307f21b484ca5f764da67728fe0c094e9ac12b82f6a2d25 Oct 31 04:14:58.279216 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:58.279177 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" event={"ID":"80b08c34-5174-4020-be20-124fad799d14","Type":"ContainerStarted","Data":"1b9367b65613af007307f21b484ca5f764da67728fe0c094e9ac12b82f6a2d25"} Oct 31 04:14:59.286353 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:59.286309 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" event={"ID":"80b08c34-5174-4020-be20-124fad799d14","Type":"ContainerStarted","Data":"9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4"} Oct 31 04:14:59.286778 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:59.286407 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:14:59.404889 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:14:59.404816 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" podStartSLOduration=2.458201986 podStartE2EDuration="3.404796354s" totalImagesPullingTime="946.594368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:59.403683134 +0000 UTC m=+3327.200401409" watchObservedRunningTime="2026-10-31 04:14:59.404796354 +0000 UTC m=+3327.201514610" Oct 31 04:15:00.159398 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.159361 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv"] Oct 31 04:15:00.162996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.162977 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.215900 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.215865 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:15:00.267888 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.267858 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv"] Oct 31 04:15:00.305314 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.305274 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.305744 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.305339 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khw5w\" (UniqueName: \"kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w\") pod \"maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.406461 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.406411 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khw5w\" (UniqueName: \"kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w\") pod \"maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.406735 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.406706 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.407376 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.407354 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.468055 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.467971 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khw5w\" (UniqueName: \"kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w\") pod \"maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.476826 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.476797 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:00.851939 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:00.851898 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv"] Oct 31 04:15:00.854530 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:15:00.854502 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394ed68f_7535_41de_8a3d_34afda3e2003.slice/crio-773f6811296b405e8912a1f257dc58637faf1b699b5887695af2166fc29f5cf8 WatchSource:0}: Error finding container 773f6811296b405e8912a1f257dc58637faf1b699b5887695af2166fc29f5cf8: Status 404 returned error can't find the container with id 773f6811296b405e8912a1f257dc58637faf1b699b5887695af2166fc29f5cf8 Oct 31 04:15:01.301121 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:01.301064 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" event={"ID":"394ed68f-7535-41de-8a3d-34afda3e2003","Type":"ContainerStarted","Data":"6905ede89129257c4a9143a7ea2270c32d0d3a06aa8e579c4e0515fa66976203"} Oct 31 04:15:01.301347 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:01.301136 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" event={"ID":"394ed68f-7535-41de-8a3d-34afda3e2003","Type":"ContainerStarted","Data":"773f6811296b405e8912a1f257dc58637faf1b699b5887695af2166fc29f5cf8"} Oct 31 04:15:01.417224 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:01.417114 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" podStartSLOduration=1.41707887 podStartE2EDuration="1.41707887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:01.415448914 +0000 UTC m=+3329.212167178" watchObservedRunningTime="2026-10-31 04:15:01.41707887 +0000 UTC m=+3329.213797128" Oct 31 04:15:02.141411 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:02.141375 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:02.320680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:02.320647 4488 generic.go:356] "Generic (PLEG): container finished" podID="394ed68f-7535-41de-8a3d-34afda3e2003" containerID="6905ede89129257c4a9143a7ea2270c32d0d3a06aa8e579c4e0515fa66976203" exitCode=7 Oct 31 04:15:02.320910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:02.320696 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" event={"ID":"394ed68f-7535-41de-8a3d-34afda3e2003","Type":"ContainerDied","Data":"6905ede89129257c4a9143a7ea2270c32d0d3a06aa8e579c4e0515fa66976203"} Oct 31 04:15:02.321035 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:02.321016 4488 scope.go:122] "RemoveContainer" containerID="6905ede89129257c4a9143a7ea2270c32d0d3a06aa8e579c4e0515fa66976203" Oct 31 04:15:03.327811 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:03.327775 4488 generic.go:356] "Generic (PLEG): container finished" podID="394ed68f-7535-41de-8a3d-34afda3e2003" containerID="e703ab24bb0419549060feb9280728d91033c77c8536dcfed817f1a68fe2c132" exitCode=7 Oct 31 04:15:03.328279 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:03.327855 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" event={"ID":"394ed68f-7535-41de-8a3d-34afda3e2003","Type":"ContainerDied","Data":"e703ab24bb0419549060feb9280728d91033c77c8536dcfed817f1a68fe2c132"} Oct 31 04:15:03.328279 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:03.327936 4488 scope.go:122] "RemoveContainer" containerID="6905ede89129257c4a9143a7ea2270c32d0d3a06aa8e579c4e0515fa66976203" Oct 31 04:15:03.328279 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:03.328215 4488 scope.go:122] "RemoveContainer" containerID="e703ab24bb0419549060feb9280728d91033c77c8536dcfed817f1a68fe2c132" Oct 31 04:15:03.328497 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:15:03.328474 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv_odh-ai-gateway-infra(394ed68f-7535-41de-8a3d-34afda3e2003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" Oct 31 04:15:04.334315 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:04.334286 4488 scope.go:122] "RemoveContainer" containerID="e703ab24bb0419549060feb9280728d91033c77c8536dcfed817f1a68fe2c132" Oct 31 04:15:04.334973 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:15:04.334537 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv_odh-ai-gateway-infra(394ed68f-7535-41de-8a3d-34afda3e2003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" Oct 31 04:15:04.887174 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:04.887132 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:15:05.297617 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:05.297543 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:15:06.268924 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:06.268891 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:15:06.589379 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:15:06.589342 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:15:06.919072 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:06.918988 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:07.277382 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:07.277296 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:15:11.781197 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:11.781157 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:15.478826 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:15.478787 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:16.133404 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:16.133367 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:16.287344 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:16.287313 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:16.309782 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:16.309752 4488 scope.go:122] "RemoveContainer" containerID="e703ab24bb0419549060feb9280728d91033c77c8536dcfed817f1a68fe2c132" Oct 31 04:15:16.941746 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:16.941709 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:17.395524 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:17.395492 4488 generic.go:356] "Generic (PLEG): container finished" podID="394ed68f-7535-41de-8a3d-34afda3e2003" containerID="4394b4ed68b7e9424c6402f1d7bcf65098d6868b91b8dac5e4d3ab023ef993ca" exitCode=60 Oct 31 04:15:17.395739 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:17.395570 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" event={"ID":"394ed68f-7535-41de-8a3d-34afda3e2003","Type":"ContainerDied","Data":"4394b4ed68b7e9424c6402f1d7bcf65098d6868b91b8dac5e4d3ab023ef993ca"} Oct 31 04:15:17.395739 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:17.395669 4488 scope.go:122] "RemoveContainer" containerID="e703ab24bb0419549060feb9280728d91033c77c8536dcfed817f1a68fe2c132" Oct 31 04:15:17.396068 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:17.396045 4488 scope.go:122] "RemoveContainer" containerID="4394b4ed68b7e9424c6402f1d7bcf65098d6868b91b8dac5e4d3ab023ef993ca" Oct 31 04:15:17.396375 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:15:17.396353 4488 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv_odh-ai-gateway-infra(394ed68f-7535-41de-8a3d-34afda3e2003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" Oct 31 04:15:18.523157 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.523121 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv"] Oct 31 04:15:18.671193 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.671164 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:18.790804 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.790720 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w\" (UniqueName: \"kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w\") pod \"394ed68f-7535-41de-8a3d-34afda3e2003\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " Oct 31 04:15:18.790977 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.790834 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca\") pod \"394ed68f-7535-41de-8a3d-34afda3e2003\" (UID: \"394ed68f-7535-41de-8a3d-34afda3e2003\") " Oct 31 04:15:18.791243 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.791210 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca" pod "394ed68f-7535-41de-8a3d-34afda3e2003" (UID: "394ed68f-7535-41de-8a3d-34afda3e2003"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:18.793279 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.793251 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w" pod "394ed68f-7535-41de-8a3d-34afda3e2003" (UID: "394ed68f-7535-41de-8a3d-34afda3e2003"). InnerVolumeSpecName "kube-api-access-khw5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:18.891863 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.891824 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khw5w\" (UniqueName: \"kubernetes.io/projected/394ed68f-7535-41de-8a3d-34afda3e2003-kube-api-access-khw5w\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:15:18.891863 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:18.891856 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ed68f-7535-41de-8a3d-34afda3e2003-maas-api-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:15:19.409805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:19.409774 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" Oct 31 04:15:19.409805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:19.409796 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv" event={"ID":"394ed68f-7535-41de-8a3d-34afda3e2003","Type":"ContainerDied","Data":"773f6811296b405e8912a1f257dc58637faf1b699b5887695af2166fc29f5cf8"} Oct 31 04:15:19.410117 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:19.409864 4488 scope.go:122] "RemoveContainer" containerID="4394b4ed68b7e9424c6402f1d7bcf65098d6868b91b8dac5e4d3ab023ef993ca" Oct 31 04:15:19.573944 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:19.573903 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv"] Oct 31 04:15:19.623668 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:19.623628 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-09c573-2989033k27sv"] Oct 31 04:15:20.315309 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:20.315275 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" path="/var/lib/kubelet/pods/394ed68f-7535-41de-8a3d-34afda3e2003/volumes" Oct 31 04:15:20.754490 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:20.754453 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:21.886537 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:21.886489 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:25.358051 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:25.358012 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:15:26.874939 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:26.874894 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:29.287224 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:29.287192 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:29.449228 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:29.449183 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:31.681514 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:31.681466 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:36.015528 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:36.015490 4488 scope.go:122] "RemoveContainer" containerID="42af9bdac56e1535b66f935b7c1699ab2ad5210878d12f5205b0954bc585dc09" Oct 31 04:15:36.024301 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:36.024281 4488 scope.go:122] "RemoveContainer" containerID="dac356b883765e5b9f822a2b1f407ca76c0d2cf1f3d344a4910c731a85312f73" Oct 31 04:15:36.032684 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:36.032659 4488 scope.go:122] "RemoveContainer" containerID="ec8b3b9f99d455b33833b603a8244cba906714ee65ebfbb636a55639df1445f6" Oct 31 04:15:36.285024 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:36.284923 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:36.578248 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:36.578214 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:36.589840 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:15:36.589805 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:15:41.401320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:41.401290 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:46.287784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:46.287709 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:51.204677 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:51.204641 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:51.787037 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:51.786991 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:15:55.984488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:15:55.984451 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:00.785677 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:00.785645 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:05.587305 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:05.587270 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:06.590015 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:16:06.589979 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:16:10.385443 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:10.385413 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:15.188817 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:15.188781 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:15.613125 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:15.613061 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:17.500430 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:17.500395 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:16:20.387533 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:20.387456 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:20.722393 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:20.722271 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:25.290746 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:25.290709 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:27.586749 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:27.586710 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5"] Oct 31 04:16:27.693598 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:27.693556 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l"] Oct 31 04:16:27.954440 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:27.954354 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:28.105171 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:28.105138 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:29.990407 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:29.990367 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:32.587064 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:32.587010 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerName="payload-processing" containerID="cri-o://f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833" gracePeriod=25 Oct 31 04:16:32.693906 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:32.693836 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" containerName="payload-pre-processing" containerID="cri-o://f7b2672a1cb87540aca448123082f6a4e972f6598e533da0130f5a865f857937" gracePeriod=25 Oct 31 04:16:34.085728 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:34.085693 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7"] Oct 31 04:16:34.086254 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:34.085907 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" podUID="80b08c34-5174-4020-be20-124fad799d14" containerName="maas-api" containerID="cri-o://9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4" gracePeriod=30 Oct 31 04:16:34.201616 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:34.201580 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:35.337154 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.337126 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:16:35.475740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.475651 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\") pod \"80b08c34-5174-4020-be20-124fad799d14\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " Oct 31 04:16:35.475740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.475702 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca\") pod \"80b08c34-5174-4020-be20-124fad799d14\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " Oct 31 04:16:35.475740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.475726 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487\" (UniqueName: \"kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487\") pod \"80b08c34-5174-4020-be20-124fad799d14\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " Oct 31 04:16:35.476101 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.475843 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs\") pod \"80b08c34-5174-4020-be20-124fad799d14\" (UID: \"80b08c34-5174-4020-be20-124fad799d14\") " Oct 31 04:16:35.476181 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.476158 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca" pod "80b08c34-5174-4020-be20-124fad799d14" (UID: "80b08c34-5174-4020-be20-124fad799d14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:35.476512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.476342 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b08c34-5174-4020-be20-124fad799d14-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:35.478336 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.478309 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487" pod "80b08c34-5174-4020-be20-124fad799d14" (UID: "80b08c34-5174-4020-be20-124fad799d14"). InnerVolumeSpecName "kube-api-access-2l487". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:35.478649 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.478630 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls" pod "80b08c34-5174-4020-be20-124fad799d14" (UID: "80b08c34-5174-4020-be20-124fad799d14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:35.478710 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.478657 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs" pod "80b08c34-5174-4020-be20-124fad799d14" (UID: "80b08c34-5174-4020-be20-124fad799d14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:16:35.576996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.576936 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:35.576996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.576989 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l487\" (UniqueName: \"kubernetes.io/projected/80b08c34-5174-4020-be20-124fad799d14-kube-api-access-2l487\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:35.576996 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.577001 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b08c34-5174-4020-be20-124fad799d14-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:35.754412 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.754300 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:35.812498 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.812461 4488 generic.go:356] "Generic (PLEG): container finished" podID="80b08c34-5174-4020-be20-124fad799d14" containerID="9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4" exitCode=0 Oct 31 04:16:35.812718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.812535 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" Oct 31 04:16:35.812718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.812553 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" event={"ID":"80b08c34-5174-4020-be20-124fad799d14","Type":"ContainerDied","Data":"9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4"} Oct 31 04:16:35.812718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.812612 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" event={"ID":"80b08c34-5174-4020-be20-124fad799d14","Type":"ContainerDied","Data":"1b9367b65613af007307f21b484ca5f764da67728fe0c094e9ac12b82f6a2d25"} Oct 31 04:16:35.812718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.812635 4488 scope.go:122] "RemoveContainer" containerID="9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4" Oct 31 04:16:35.822603 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.822581 4488 scope.go:122] "RemoveContainer" containerID="9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4" Oct 31 04:16:35.822943 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:16:35.822923 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4\": container with ID starting with 9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4 not found: ID does not exist" containerID="9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4" Oct 31 04:16:35.822992 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.822954 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4"} err="failed to get container status \"9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4\": rpc error: code = NotFound desc = could not find container \"9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4\": container with ID starting with 9cab5bc5c55ab98b92bcd2d8b131b38bcd052c0218ab7dcb1b0141cf8c6b8ee4 not found: ID does not exist" Oct 31 04:16:35.981035 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:35.980994 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7"] Oct 31 04:16:36.030333 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:36.030235 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7"] Oct 31 04:16:36.052852 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:36.052818 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:36.266166 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:36.266130 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:16:36.315588 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:36.315503 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b08c34-5174-4020-be20-124fad799d14" path="/var/lib/kubelet/pods/80b08c34-5174-4020-be20-124fad799d14/volumes" Oct 31 04:16:36.589122 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:16:36.589062 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:16:36.628178 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:36.628139 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:36.826809 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:36.826774 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:37.275767 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:37.275733 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:16:38.292632 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:38.292584 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-09c573-f85dd556c-fgcw7" podUID="80b08c34-5174-4020-be20-124fad799d14" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.219:8443/health\": context deadline exceeded" Oct 31 04:16:38.962953 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:38.962912 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:16:42.719430 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:42.719395 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:42.873310 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:42.873276 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:43.065880 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:43.065792 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:46.266197 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:46.266156 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:16:47.275412 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:47.275371 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:16:50.885415 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:50.885381 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:52.834682 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.834653 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:16:52.895201 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.895168 4488 generic.go:356] "Generic (PLEG): container finished" podID="abd22191-42e5-47f7-bd46-61b046d458b4" containerID="f7b2672a1cb87540aca448123082f6a4e972f6598e533da0130f5a865f857937" exitCode=0 Oct 31 04:16:52.895397 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.895217 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" event={"ID":"abd22191-42e5-47f7-bd46-61b046d458b4","Type":"ContainerDied","Data":"f7b2672a1cb87540aca448123082f6a4e972f6598e533da0130f5a865f857937"} Oct 31 04:16:52.896894 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.896870 4488 generic.go:356] "Generic (PLEG): container finished" podID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerID="f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833" exitCode=0 Oct 31 04:16:52.897074 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.896923 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" Oct 31 04:16:52.897074 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.896933 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" event={"ID":"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47","Type":"ContainerDied","Data":"f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833"} Oct 31 04:16:52.897074 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.896969 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5" event={"ID":"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47","Type":"ContainerDied","Data":"3adb640e999db84389eb4ec58f3c10a890bcf1281ab48d07b72b2ae81cae80bd"} Oct 31 04:16:52.897074 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.896991 4488 scope.go:122] "RemoveContainer" containerID="f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833" Oct 31 04:16:52.906398 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.906370 4488 scope.go:122] "RemoveContainer" containerID="f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833" Oct 31 04:16:52.906803 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:16:52.906779 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833\": container with ID starting with f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833 not found: ID does not exist" containerID="f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833" Oct 31 04:16:52.906931 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.906815 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833"} err="failed to get container status \"f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833\": rpc error: code = NotFound desc = could not find container \"f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833\": container with ID starting with f64f9b63f67f69c7d0a0175a67ea0502da6c5965fc30d74963cbb2e11f91f833 not found: ID does not exist" Oct 31 04:16:52.944807 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.944783 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:16:52.945701 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.945680 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume\") pod \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " Oct 31 04:16:52.945773 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.945761 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d\" (UniqueName: \"kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d\") pod \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\" (UID: \"aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47\") " Oct 31 04:16:52.946038 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.946020 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume" pod "aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" (UID: "aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:52.947981 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:52.947952 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d" pod "aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" (UID: "aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47"). InnerVolumeSpecName "kube-api-access-dqs5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:53.047259 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.047224 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume\") pod \"abd22191-42e5-47f7-bd46-61b046d458b4\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " Oct 31 04:16:53.047259 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.047267 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v\" (UniqueName: \"kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v\") pod \"abd22191-42e5-47f7-bd46-61b046d458b4\" (UID: \"abd22191-42e5-47f7-bd46-61b046d458b4\") " Oct 31 04:16:53.047566 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.047545 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:53.047611 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.047573 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqs5d\" (UniqueName: \"kubernetes.io/projected/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47-kube-api-access-dqs5d\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:53.047664 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.047635 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume" pod "abd22191-42e5-47f7-bd46-61b046d458b4" (UID: "abd22191-42e5-47f7-bd46-61b046d458b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:16:53.049835 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.049811 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v" pod "abd22191-42e5-47f7-bd46-61b046d458b4" (UID: "abd22191-42e5-47f7-bd46-61b046d458b4"). InnerVolumeSpecName "kube-api-access-h227v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:16:53.148565 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.148474 4488 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd22191-42e5-47f7-bd46-61b046d458b4-plugins-config-volume\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:53.148565 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.148507 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h227v\" (UniqueName: \"kubernetes.io/projected/abd22191-42e5-47f7-bd46-61b046d458b4-kube-api-access-h227v\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:16:53.321271 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.321230 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5"] Oct 31 04:16:53.325995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.325965 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-09c573-689db9cd442zt5"] Oct 31 04:16:53.903278 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.903190 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" Oct 31 04:16:53.903278 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.903215 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l" event={"ID":"abd22191-42e5-47f7-bd46-61b046d458b4","Type":"ContainerDied","Data":"7d5991dfde8f4cc6fdf0f5b5ad6697fcf08778fa5e4b4b3a2730ab01382f90cf"} Oct 31 04:16:53.903864 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:53.903288 4488 scope.go:122] "RemoveContainer" containerID="f7b2672a1cb87540aca448123082f6a4e972f6598e533da0130f5a865f857937" Oct 31 04:16:54.067548 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:54.067511 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l"] Oct 31 04:16:54.072079 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:54.072048 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-09c573-c8549jnz2l"] Oct 31 04:16:54.315047 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:54.314961 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" path="/var/lib/kubelet/pods/aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47/volumes" Oct 31 04:16:54.315400 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:54.315385 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" path="/var/lib/kubelet/pods/abd22191-42e5-47f7-bd46-61b046d458b4/volumes" Oct 31 04:16:55.304894 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:55.304855 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:16:57.401668 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:57.401622 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:58.323825 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:16:58.323788 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:00.046909 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:00.046875 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2"] Oct 31 04:17:00.096560 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:00.096519 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2n4r2"] Oct 31 04:17:00.315010 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:00.314921 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ace00f-8f79-4c36-b460-1987ccabc1fa" path="/var/lib/kubelet/pods/07ace00f-8f79-4c36-b460-1987ccabc1fa/volumes" Oct 31 04:17:06.589033 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:17:06.588988 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:17:12.720963 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:12.720932 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:34.159635 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:34.159579 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:36.588778 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:17:36.588739 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:17:36.984396 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:36.984307 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:38.812876 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:38.812834 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:17:41.627116 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:41.623215 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:17:41.627116 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:41.623314 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:17:51.688051 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:17:51.688008 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:18:06.588846 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:18:06.588803 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:18:07.292059 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292017 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4"] Oct 31 04:18:07.292407 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292392 4488 cpu_manager.go:414] "containerMap: removing container" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.292407 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292407 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292421 4488 cpu_manager.go:414] "containerMap: removing container" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerName="payload-processing" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292426 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerName="payload-processing" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292435 4488 cpu_manager.go:414] "containerMap: removing container" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292440 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292453 4488 cpu_manager.go:414] "containerMap: removing container" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" containerName="payload-pre-processing" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292458 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" containerName="payload-pre-processing" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292480 4488 cpu_manager.go:414] "containerMap: removing container" podUID="80b08c34-5174-4020-be20-124fad799d14" containerName="maas-api" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292485 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b08c34-5174-4020-be20-124fad799d14" containerName="maas-api" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292538 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd22191-42e5-47f7-bd46-61b046d458b4" containerName="payload-pre-processing" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292549 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.292559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292559 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b08c34-5174-4020-be20-124fad799d14" containerName="maas-api" Oct 31 04:18:07.292985 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292569 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.292985 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292577 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa50cc2a-ce0c-40d6-b0ae-75a8f2e3ec47" containerName="payload-processing" Oct 31 04:18:07.292985 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.292582 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:07.295447 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.295430 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.347715 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.347677 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:18:07.347715 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.347701 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:18:07.347715 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.347714 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:18:07.397712 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.397679 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:18:07.399379 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.399354 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4"] Oct 31 04:18:07.452962 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.452922 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.453230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.453116 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.453230 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.453197 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptkq\" (UniqueName: \"kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.554006 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.553893 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.554006 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.553966 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.554274 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.554030 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptkq\" (UniqueName: \"kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.554587 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.554567 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.556518 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.556494 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.615412 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.615383 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptkq\" (UniqueName: \"kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq\") pod \"maas-api-revoke-sub-452b6d7b-5wqs4\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:07.908852 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:07.908812 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:08.283650 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:08.283555 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4"] Oct 31 04:18:08.286677 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:18:08.286648 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19031f15_7bc1_4377_9730_a3368593113d.slice/crio-016b69054302791aba0775518e2548dc7b48c638bcc736778da01bc5b4358efe WatchSource:0}: Error finding container 016b69054302791aba0775518e2548dc7b48c638bcc736778da01bc5b4358efe: Status 404 returned error can't find the container with id 016b69054302791aba0775518e2548dc7b48c638bcc736778da01bc5b4358efe Oct 31 04:18:08.322304 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:08.322272 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" event={"ID":"19031f15-7bc1-4377-9730-a3368593113d","Type":"ContainerStarted","Data":"016b69054302791aba0775518e2548dc7b48c638bcc736778da01bc5b4358efe"} Oct 31 04:18:08.492949 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:08.492911 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:18:09.335211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:09.334739 4488 generic.go:356] "Generic (PLEG): container finished" podID="19031f15-7bc1-4377-9730-a3368593113d" containerID="6e6f5dc64f81a8e3f5ebae0a6ba296310af038c134d5367d526d7d3f0cfcbce3" exitCode=0 Oct 31 04:18:09.335211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:09.334789 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" event={"ID":"19031f15-7bc1-4377-9730-a3368593113d","Type":"ContainerDied","Data":"6e6f5dc64f81a8e3f5ebae0a6ba296310af038c134d5367d526d7d3f0cfcbce3"} Oct 31 04:18:11.018637 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.018602 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:11.072115 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.071672 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca\") pod \"19031f15-7bc1-4377-9730-a3368593113d\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " Oct 31 04:18:11.072115 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.071780 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca\") pod \"19031f15-7bc1-4377-9730-a3368593113d\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " Oct 31 04:18:11.072115 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.071868 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq\" (UniqueName: \"kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq\") pod \"19031f15-7bc1-4377-9730-a3368593113d\" (UID: \"19031f15-7bc1-4377-9730-a3368593113d\") " Oct 31 04:18:11.073980 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.073949 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca" pod "19031f15-7bc1-4377-9730-a3368593113d" (UID: "19031f15-7bc1-4377-9730-a3368593113d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:11.103118 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.094238 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca" pod "19031f15-7bc1-4377-9730-a3368593113d" (UID: "19031f15-7bc1-4377-9730-a3368593113d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:11.103118 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.094887 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq" pod "19031f15-7bc1-4377-9730-a3368593113d" (UID: "19031f15-7bc1-4377-9730-a3368593113d"). InnerVolumeSpecName "kube-api-access-wptkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:11.185176 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.184297 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:11.185176 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.184371 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:11.190248 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.190189 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptkq\" (UniqueName: \"kubernetes.io/projected/19031f15-7bc1-4377-9730-a3368593113d-kube-api-access-wptkq\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:11.190555 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.190533 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19031f15-7bc1-4377-9730-a3368593113d-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:11.190702 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.190687 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19031f15-7bc1-4377-9730-a3368593113d-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:11.197425 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.197261 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:11.197425 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.197350 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:11.427460 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.427417 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" event={"ID":"19031f15-7bc1-4377-9730-a3368593113d","Type":"ContainerDied","Data":"016b69054302791aba0775518e2548dc7b48c638bcc736778da01bc5b4358efe"} Oct 31 04:18:11.427749 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.427731 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016b69054302791aba0775518e2548dc7b48c638bcc736778da01bc5b4358efe" Oct 31 04:18:11.427948 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.427921 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4" Oct 31 04:18:11.624739 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.624219 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:11.624739 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:11.624292 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:20.194880 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:20.194840 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:18:23.203791 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:23.203749 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:18:25.736291 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:25.736232 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-znzrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:25.736985 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:25.736925 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-znzrz" podUID="7f7842f2-5eeb-417d-8bcb-76f99f23fe23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:26.188320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:26.188246 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:26.188659 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:26.188625 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:26.215261 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:26.215217 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:26.215556 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:26.215527 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:26.615130 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:26.614214 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:18:26.615130 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:26.614340 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:18:32.763328 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:32.763279 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:18:36.090070 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.090037 4488 scope.go:122] "RemoveContainer" containerID="a4445699e3e4631f789882cfe26d8cd7948d3f88cc839259892751a539af63a5" Oct 31 04:18:36.594583 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:18:36.594547 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:18:36.996348 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.996265 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq"] Oct 31 04:18:36.996631 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.996619 4488 cpu_manager.go:414] "containerMap: removing container" podUID="19031f15-7bc1-4377-9730-a3368593113d" containerName="revoke-keys" Oct 31 04:18:36.996680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.996632 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19031f15-7bc1-4377-9730-a3368593113d" containerName="revoke-keys" Oct 31 04:18:36.996680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.996657 4488 cpu_manager.go:414] "containerMap: removing container" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:36.996680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.996662 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394ed68f-7535-41de-8a3d-34afda3e2003" containerName="cleanup" Oct 31 04:18:36.996805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.996721 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="19031f15-7bc1-4377-9730-a3368593113d" containerName="revoke-keys" Oct 31 04:18:36.999693 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:36.999676 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.062196 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.062161 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:18:37.111441 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.111403 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:18:37.111870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.111494 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:18:37.111870 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.111559 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:18:37.113591 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.113571 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq"] Oct 31 04:18:37.130612 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.130578 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.130784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.130646 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.130784 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.130711 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw5nv\" (UniqueName: \"kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.231951 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.231900 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.232189 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.231998 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.232189 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.232025 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw5nv\" (UniqueName: \"kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.232687 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.232667 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.234527 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.234503 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.294054 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.293977 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw5nv\" (UniqueName: \"kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv\") pod \"maas-api-revoke-sub-3e31a506-fgnqq\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.312863 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.312829 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:37.442407 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.442374 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:18:37.514438 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.514397 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:18:37.687290 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:37.687254 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq"] Oct 31 04:18:37.689432 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:18:37.689401 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2510e7d2_caea_4c22_b8ca_152f99864799.slice/crio-99dc3779732a374c5ecf8e973467db2e009fe5e9f5c5c1bce350f99693af1f38 WatchSource:0}: Error finding container 99dc3779732a374c5ecf8e973467db2e009fe5e9f5c5c1bce350f99693af1f38: Status 404 returned error can't find the container with id 99dc3779732a374c5ecf8e973467db2e009fe5e9f5c5c1bce350f99693af1f38 Oct 31 04:18:38.562789 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:38.562682 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" event={"ID":"2510e7d2-caea-4c22-b8ca-152f99864799","Type":"ContainerStarted","Data":"229e7bb987450036ee296a08f7434d2be67a051c1ea45923b30ffac4fdef03e3"} Oct 31 04:18:38.562789 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:38.562753 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" event={"ID":"2510e7d2-caea-4c22-b8ca-152f99864799","Type":"ContainerStarted","Data":"99dc3779732a374c5ecf8e973467db2e009fe5e9f5c5c1bce350f99693af1f38"} Oct 31 04:18:38.680397 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:38.680327 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" podStartSLOduration=2.680307944 podStartE2EDuration="2.680307944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:38.677872543 +0000 UTC m=+3546.474590802" watchObservedRunningTime="2026-10-31 04:18:38.680307944 +0000 UTC m=+3546.477026204" Oct 31 04:18:39.585123 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:39.580904 4488 generic.go:356] "Generic (PLEG): container finished" podID="2510e7d2-caea-4c22-b8ca-152f99864799" containerID="229e7bb987450036ee296a08f7434d2be67a051c1ea45923b30ffac4fdef03e3" exitCode=0 Oct 31 04:18:39.585123 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:39.581068 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" event={"ID":"2510e7d2-caea-4c22-b8ca-152f99864799","Type":"ContainerDied","Data":"229e7bb987450036ee296a08f7434d2be67a051c1ea45923b30ffac4fdef03e3"} Oct 31 04:18:40.745994 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.745968 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:40.814595 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.814559 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca\") pod \"2510e7d2-caea-4c22-b8ca-152f99864799\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " Oct 31 04:18:40.814790 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.814616 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca\") pod \"2510e7d2-caea-4c22-b8ca-152f99864799\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " Oct 31 04:18:40.814790 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.814636 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv\" (UniqueName: \"kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv\") pod \"2510e7d2-caea-4c22-b8ca-152f99864799\" (UID: \"2510e7d2-caea-4c22-b8ca-152f99864799\") " Oct 31 04:18:40.814938 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.814916 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca" pod "2510e7d2-caea-4c22-b8ca-152f99864799" (UID: "2510e7d2-caea-4c22-b8ca-152f99864799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:18:40.816973 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.816943 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca" pod "2510e7d2-caea-4c22-b8ca-152f99864799" (UID: "2510e7d2-caea-4c22-b8ca-152f99864799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:18:40.817133 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.817030 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv" pod "2510e7d2-caea-4c22-b8ca-152f99864799" (UID: "2510e7d2-caea-4c22-b8ca-152f99864799"). InnerVolumeSpecName "kube-api-access-pw5nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:18:40.915654 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.915562 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:40.915654 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.915596 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2510e7d2-caea-4c22-b8ca-152f99864799-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:40.915654 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:40.915607 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw5nv\" (UniqueName: \"kubernetes.io/projected/2510e7d2-caea-4c22-b8ca-152f99864799-kube-api-access-pw5nv\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:18:41.593244 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:41.593215 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" Oct 31 04:18:41.593436 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:41.593242 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq" event={"ID":"2510e7d2-caea-4c22-b8ca-152f99864799","Type":"ContainerDied","Data":"99dc3779732a374c5ecf8e973467db2e009fe5e9f5c5c1bce350f99693af1f38"} Oct 31 04:18:41.593436 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:41.593284 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99dc3779732a374c5ecf8e973467db2e009fe5e9f5c5c1bce350f99693af1f38" Oct 31 04:18:49.691664 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:18:49.691627 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:19:00.371336 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:00.371293 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:19:06.588349 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:19:06.588311 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:19:08.047165 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.047126 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4"] Oct 31 04:19:08.097419 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.097382 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kb2k4"] Oct 31 04:19:08.098919 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.098895 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852"] Oct 31 04:19:08.099332 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.099318 4488 cpu_manager.go:414] "containerMap: removing container" podUID="2510e7d2-caea-4c22-b8ca-152f99864799" containerName="revoke-keys" Oct 31 04:19:08.099380 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.099334 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2510e7d2-caea-4c22-b8ca-152f99864799" containerName="revoke-keys" Oct 31 04:19:08.099417 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.099409 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="2510e7d2-caea-4c22-b8ca-152f99864799" containerName="revoke-keys" Oct 31 04:19:08.102553 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.102536 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.157319 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.157290 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:19:08.157565 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.157353 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:08.157683 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.157404 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:08.157757 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.157519 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\"" Oct 31 04:19:08.159537 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.159513 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852"] Oct 31 04:19:08.161134 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.161078 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.161265 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.161192 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvl9\" (UniqueName: \"kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.161265 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.161256 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.262712 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.262666 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.262712 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.262711 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvl9\" (UniqueName: \"kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.263006 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.262751 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.263381 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.263359 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.265295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.265271 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.316382 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.316306 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7842fcf2-e087-4208-8ed6-37334c98e188" path="/var/lib/kubelet/pods/7842fcf2-e087-4208-8ed6-37334c98e188/volumes" Oct 31 04:19:08.324048 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.324018 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvl9\" (UniqueName: \"kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9\") pod \"maas-api-revoke-sub-53c072c2-wr852\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.415894 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.415856 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:08.792064 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:08.792039 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852"] Oct 31 04:19:08.794635 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:19:08.794603 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50f9439_8dd3_40f0_8393_acb83b97e24b.slice/crio-ac018f673708f372bdd18d04a07a728cbef00b48732a9471509b0a6a2f891b8b WatchSource:0}: Error finding container ac018f673708f372bdd18d04a07a728cbef00b48732a9471509b0a6a2f891b8b: Status 404 returned error can't find the container with id ac018f673708f372bdd18d04a07a728cbef00b48732a9471509b0a6a2f891b8b Oct 31 04:19:09.752534 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:09.752504 4488 generic.go:356] "Generic (PLEG): container finished" podID="f50f9439-8dd3-40f0-8393-acb83b97e24b" containerID="da33c79cfd23f76d3b90890f3a53f5b3146d5cea7d3c29fbedcc44a8a12ce2dc" exitCode=0 Oct 31 04:19:09.752999 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:09.752558 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" event={"ID":"f50f9439-8dd3-40f0-8393-acb83b97e24b","Type":"ContainerDied","Data":"da33c79cfd23f76d3b90890f3a53f5b3146d5cea7d3c29fbedcc44a8a12ce2dc"} Oct 31 04:19:09.752999 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:09.752603 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" event={"ID":"f50f9439-8dd3-40f0-8393-acb83b97e24b","Type":"ContainerStarted","Data":"ac018f673708f372bdd18d04a07a728cbef00b48732a9471509b0a6a2f891b8b"} Oct 31 04:19:10.900670 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.900648 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:10.988367 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.988336 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9\" (UniqueName: \"kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9\") pod \"f50f9439-8dd3-40f0-8393-acb83b97e24b\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " Oct 31 04:19:10.988566 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.988409 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca\") pod \"f50f9439-8dd3-40f0-8393-acb83b97e24b\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " Oct 31 04:19:10.988566 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.988452 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca\") pod \"f50f9439-8dd3-40f0-8393-acb83b97e24b\" (UID: \"f50f9439-8dd3-40f0-8393-acb83b97e24b\") " Oct 31 04:19:10.988883 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.988859 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca" pod "f50f9439-8dd3-40f0-8393-acb83b97e24b" (UID: "f50f9439-8dd3-40f0-8393-acb83b97e24b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:10.990612 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.990578 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9" pod "f50f9439-8dd3-40f0-8393-acb83b97e24b" (UID: "f50f9439-8dd3-40f0-8393-acb83b97e24b"). InnerVolumeSpecName "kube-api-access-stvl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:10.990757 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:10.990698 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca" pod "f50f9439-8dd3-40f0-8393-acb83b97e24b" (UID: "f50f9439-8dd3-40f0-8393-acb83b97e24b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:11.089540 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:11.089506 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvl9\" (UniqueName: \"kubernetes.io/projected/f50f9439-8dd3-40f0-8393-acb83b97e24b-kube-api-access-stvl9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:11.089540 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:11.089537 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-xks-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:11.089540 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:11.089548 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50f9439-8dd3-40f0-8393-acb83b97e24b-maas-api-openshift-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:11.763031 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:11.763004 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" Oct 31 04:19:11.763031 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:11.763022 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852" event={"ID":"f50f9439-8dd3-40f0-8393-acb83b97e24b","Type":"ContainerDied","Data":"ac018f673708f372bdd18d04a07a728cbef00b48732a9471509b0a6a2f891b8b"} Oct 31 04:19:11.763293 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:11.763066 4488 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac018f673708f372bdd18d04a07a728cbef00b48732a9471509b0a6a2f891b8b" Oct 31 04:19:20.876895 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:20.876859 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:19:21.301720 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.301628 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:21.407730 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.407698 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8"] Oct 31 04:19:21.408131 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.408113 4488 cpu_manager.go:414] "containerMap: removing container" podUID="f50f9439-8dd3-40f0-8393-acb83b97e24b" containerName="revoke-keys" Oct 31 04:19:21.408131 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.408129 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50f9439-8dd3-40f0-8393-acb83b97e24b" containerName="revoke-keys" Oct 31 04:19:21.408310 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.408235 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50f9439-8dd3-40f0-8393-acb83b97e24b" containerName="revoke-keys" Oct 31 04:19:21.411285 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.411249 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.469955 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.469915 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:21.491177 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.491067 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rtm\" (UniqueName: \"kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.491177 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.491170 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.491395 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.491191 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.491395 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.491340 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.513536 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.513505 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\"" Oct 31 04:19:21.513708 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.513522 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 04:19:21.513708 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.513654 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 04:19:21.513796 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.513710 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:19:21.513796 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.513725 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:19:21.515707 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.515682 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8"] Oct 31 04:19:21.592504 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.592475 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.592734 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.592531 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rtm\" (UniqueName: \"kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.592734 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.592576 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.592734 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.592597 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.593436 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.593408 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.595079 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.595051 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.595461 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.595441 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.653801 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.653764 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rtm\" (UniqueName: \"kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm\") pod \"maas-api-587c7c8bb4-qgmb8\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:21.724218 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:21.724182 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:22.098980 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:22.098953 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8"] Oct 31 04:19:22.101275 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:19:22.101248 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a587550_4eaf_4494_832c_ec8d59e692ad.slice/crio-719cfc247f59a3415515d16de16582d92ffa50d6ed72d2de149f9bd1d240fcb3 WatchSource:0}: Error finding container 719cfc247f59a3415515d16de16582d92ffa50d6ed72d2de149f9bd1d240fcb3: Status 404 returned error can't find the container with id 719cfc247f59a3415515d16de16582d92ffa50d6ed72d2de149f9bd1d240fcb3 Oct 31 04:19:22.102610 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:22.102590 4488 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:19:22.819016 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:22.818981 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" event={"ID":"6a587550-4eaf-4494-832c-ec8d59e692ad","Type":"ContainerStarted","Data":"719cfc247f59a3415515d16de16582d92ffa50d6ed72d2de149f9bd1d240fcb3"} Oct 31 04:19:23.825596 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:23.825557 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" event={"ID":"6a587550-4eaf-4494-832c-ec8d59e692ad","Type":"ContainerStarted","Data":"ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216"} Oct 31 04:19:23.826047 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:23.825685 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:23.947177 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:23.947117 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" podStartSLOduration=2.07105276 podStartE2EDuration="2.947098456s" totalImagesPullingTime="876.045696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:23.945007022 +0000 UTC m=+3591.741725282" watchObservedRunningTime="2026-10-31 04:19:23.947098456 +0000 UTC m=+3591.743816708" Oct 31 04:19:27.992376 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:27.992343 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:28.517059 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.517022 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx"] Oct 31 04:19:28.520556 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.520533 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.573750 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.573707 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a058534b-openshift-default-dockercfg-hz4g5\"" Oct 31 04:19:28.621541 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.621509 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:28.624534 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.624513 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx"] Oct 31 04:19:28.663229 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663195 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663280 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663421 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663391 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cml9\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663504 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663434 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663504 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663493 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663572 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663520 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663572 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663559 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663648 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663584 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.663648 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.663620 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.764719 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.764672 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.764719 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.764722 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765010 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.764868 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765010 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.764911 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765010 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.764940 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765010 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.764990 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765263 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765024 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765328 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765282 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765396 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765355 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765458 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765438 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765520 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765494 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.765910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765869 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.766042 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.765959 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cml9\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.766414 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.766396 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.767556 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.767497 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.767659 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.767549 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.825962 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.825933 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.826182 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.826158 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cml9\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9\") pod \"e2e-ait-a058534b-openshift-default-9657f649c-xm8rx\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.834020 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.833985 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:28.943434 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:28.943406 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:29.263033 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.262996 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx"] Oct 31 04:19:29.266150 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:19:29.266124 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf851b6_c782_4965_a644_a5e4edfdb82a.slice/crio-72a30c7b909c3990bfd2b3e6ed5195be9c6f3220259ed5495d9eb3831e39e054 WatchSource:0}: Error finding container 72a30c7b909c3990bfd2b3e6ed5195be9c6f3220259ed5495d9eb3831e39e054: Status 404 returned error can't find the container with id 72a30c7b909c3990bfd2b3e6ed5195be9c6f3220259ed5495d9eb3831e39e054 Oct 31 04:19:29.268359 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.268322 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 04:19:29.268494 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.268408 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 04:19:29.268494 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.268454 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 04:19:29.836131 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.836074 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:29.863110 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.857926 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" event={"ID":"aaf851b6-c782-4965-a644-a5e4edfdb82a","Type":"ContainerStarted","Data":"70b26b6495ea52f275224b4dc96dd031780bea5a1dc5fc00f5d53553e75d4500"} Oct 31 04:19:29.863110 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:29.858050 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" event={"ID":"aaf851b6-c782-4965-a644-a5e4edfdb82a","Type":"ContainerStarted","Data":"72a30c7b909c3990bfd2b3e6ed5195be9c6f3220259ed5495d9eb3831e39e054"} Oct 31 04:19:30.051508 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.051471 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c"] Oct 31 04:19:30.055071 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.055044 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.077451 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.077401 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podStartSLOduration=2.077386074 podStartE2EDuration="2.077386074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:30.07572694 +0000 UTC m=+3597.872445199" watchObservedRunningTime="2026-10-31 04:19:30.077386074 +0000 UTC m=+3597.874104332" Oct 31 04:19:30.119385 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.119295 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c"] Oct 31 04:19:30.181037 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.180995 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05dd293-d64e-4dac-910f-27f6569ae93f-maas-api-tls\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.181037 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.181045 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv47k\" (UniqueName: \"kubernetes.io/projected/f05dd293-d64e-4dac-910f-27f6569ae93f-kube-api-access-rv47k\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.181320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.181071 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05dd293-d64e-4dac-910f-27f6569ae93f-openshift-service-ca\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.181320 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.181231 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05dd293-d64e-4dac-910f-27f6569ae93f-metrics-certs\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.281794 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.281744 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05dd293-d64e-4dac-910f-27f6569ae93f-maas-api-tls\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.281794 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.281804 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47k\" (UniqueName: \"kubernetes.io/projected/f05dd293-d64e-4dac-910f-27f6569ae93f-kube-api-access-rv47k\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.282338 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.281828 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05dd293-d64e-4dac-910f-27f6569ae93f-openshift-service-ca\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.282338 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.281866 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05dd293-d64e-4dac-910f-27f6569ae93f-metrics-certs\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.282618 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.282599 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05dd293-d64e-4dac-910f-27f6569ae93f-openshift-service-ca\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.284462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.284433 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05dd293-d64e-4dac-910f-27f6569ae93f-maas-api-tls\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.284576 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.284470 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05dd293-d64e-4dac-910f-27f6569ae93f-metrics-certs\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.342902 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.342869 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv47k\" (UniqueName: \"kubernetes.io/projected/f05dd293-d64e-4dac-910f-27f6569ae93f-kube-api-access-rv47k\") pod \"maas-api-796fbdf5b4-ds97c\" (UID: \"f05dd293-d64e-4dac-910f-27f6569ae93f\") " pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.368812 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.368774 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:30.744154 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.744059 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c"] Oct 31 04:19:30.746940 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:19:30.746914 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05dd293_d64e_4dac_910f_27f6569ae93f.slice/crio-bdc048ae44a3f699adce53c6d80f6fbb9e1af9d811203b31ec13bc78093417a2 WatchSource:0}: Error finding container bdc048ae44a3f699adce53c6d80f6fbb9e1af9d811203b31ec13bc78093417a2: Status 404 returned error can't find the container with id bdc048ae44a3f699adce53c6d80f6fbb9e1af9d811203b31ec13bc78093417a2 Oct 31 04:19:30.834653 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.834618 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:30.839860 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.839832 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:30.866004 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.865969 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" event={"ID":"f05dd293-d64e-4dac-910f-27f6569ae93f","Type":"ContainerStarted","Data":"bdc048ae44a3f699adce53c6d80f6fbb9e1af9d811203b31ec13bc78093417a2"} Oct 31 04:19:30.866372 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.866353 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:30.867462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:30.867434 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:31.873007 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:31.872963 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" event={"ID":"f05dd293-d64e-4dac-910f-27f6569ae93f","Type":"ContainerStarted","Data":"a4c79c66849df542420f9d0504b1ea8bc46c811d4040a3bf6325aee9c850b7dd"} Oct 31 04:19:31.991828 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:31.991771 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" podStartSLOduration=1.174039185 podStartE2EDuration="1.991754235s" totalImagesPullingTime="817.71505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:31.99031414 +0000 UTC m=+3599.787032402" watchObservedRunningTime="2026-10-31 04:19:31.991754235 +0000 UTC m=+3599.788472495" Oct 31 04:19:32.303794 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:32.303705 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:32.465544 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:32.465511 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:32.621608 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:32.621560 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:32.877672 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:32.877589 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:35.480520 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:35.480489 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:35.661308 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:35.661272 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:36.589603 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:19:36.589561 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:19:36.928180 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:36.928078 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:37.050278 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:37.050238 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg"] Oct 31 04:19:37.099817 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:37.099776 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pwxlg"] Oct 31 04:19:37.128657 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:37.128618 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:37.332862 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:37.332798 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:37.674744 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:37.674652 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:38.315683 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:38.315652 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7a6179-3ad1-43ce-be1c-6427da8ddad7" path="/var/lib/kubelet/pods/ed7a6179-3ad1-43ce-be1c-6427da8ddad7/volumes" Oct 31 04:19:38.888804 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:38.888778 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796fbdf5b4-ds97c" Oct 31 04:19:39.058522 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.058489 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8"] Oct 31 04:19:39.058805 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.058777 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" podUID="6a587550-4eaf-4494-832c-ec8d59e692ad" containerName="maas-api" containerID="cri-o://ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216" gracePeriod=30 Oct 31 04:19:39.317619 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.317596 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:39.488334 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.488248 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs\") pod \"6a587550-4eaf-4494-832c-ec8d59e692ad\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " Oct 31 04:19:39.488538 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.488335 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm\" (UniqueName: \"kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm\") pod \"6a587550-4eaf-4494-832c-ec8d59e692ad\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " Oct 31 04:19:39.488538 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.488416 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca\") pod \"6a587550-4eaf-4494-832c-ec8d59e692ad\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " Oct 31 04:19:39.488538 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.488452 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls\") pod \"6a587550-4eaf-4494-832c-ec8d59e692ad\" (UID: \"6a587550-4eaf-4494-832c-ec8d59e692ad\") " Oct 31 04:19:39.488835 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.488806 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca" pod "6a587550-4eaf-4494-832c-ec8d59e692ad" (UID: "6a587550-4eaf-4494-832c-ec8d59e692ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:19:39.490804 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.490773 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm" pod "6a587550-4eaf-4494-832c-ec8d59e692ad" (UID: "6a587550-4eaf-4494-832c-ec8d59e692ad"). InnerVolumeSpecName "kube-api-access-b8rtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:19:39.490804 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.490774 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls" pod "6a587550-4eaf-4494-832c-ec8d59e692ad" (UID: "6a587550-4eaf-4494-832c-ec8d59e692ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:39.490953 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.490902 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs" pod "6a587550-4eaf-4494-832c-ec8d59e692ad" (UID: "6a587550-4eaf-4494-832c-ec8d59e692ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:19:39.589447 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.589411 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:39.589447 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.589439 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a587550-4eaf-4494-832c-ec8d59e692ad-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:39.589447 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.589449 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rtm\" (UniqueName: \"kubernetes.io/projected/6a587550-4eaf-4494-832c-ec8d59e692ad-kube-api-access-b8rtm\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:39.589447 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.589458 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a587550-4eaf-4494-832c-ec8d59e692ad-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:19:39.916817 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.916769 4488 generic.go:356] "Generic (PLEG): container finished" podID="6a587550-4eaf-4494-832c-ec8d59e692ad" containerID="ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216" exitCode=0 Oct 31 04:19:39.917295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.916887 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" Oct 31 04:19:39.917295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.916915 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" event={"ID":"6a587550-4eaf-4494-832c-ec8d59e692ad","Type":"ContainerDied","Data":"ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216"} Oct 31 04:19:39.917295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.916978 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8" event={"ID":"6a587550-4eaf-4494-832c-ec8d59e692ad","Type":"ContainerDied","Data":"719cfc247f59a3415515d16de16582d92ffa50d6ed72d2de149f9bd1d240fcb3"} Oct 31 04:19:39.917295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.917009 4488 scope.go:122] "RemoveContainer" containerID="ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216" Oct 31 04:19:39.926984 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.926964 4488 scope.go:122] "RemoveContainer" containerID="ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216" Oct 31 04:19:39.927322 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:19:39.927295 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216\": container with ID starting with ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216 not found: ID does not exist" containerID="ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216" Oct 31 04:19:39.927411 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:39.927335 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216"} err="failed to get container status \"ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216\": rpc error: code = NotFound desc = could not find container \"ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216\": container with ID starting with ad5b816c5c7fd4dc84b4f142bbbb7d84e026a96a15b81ca73544583bcbe27216 not found: ID does not exist" Oct 31 04:19:40.081481 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:40.081443 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8"] Oct 31 04:19:40.131391 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:40.131348 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-587c7c8bb4-qgmb8"] Oct 31 04:19:40.315475 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:40.315396 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a587550-4eaf-4494-832c-ec8d59e692ad" path="/var/lib/kubelet/pods/6a587550-4eaf-4494-832c-ec8d59e692ad/volumes" Oct 31 04:19:42.107583 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:42.107554 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:46.082744 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:46.082693 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:19:46.958892 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:46.958857 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:51.852156 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:51.852118 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:56.002936 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.002844 4488 generic.go:356] "Generic (PLEG): container finished" podID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerID="70b26b6495ea52f275224b4dc96dd031780bea5a1dc5fc00f5d53553e75d4500" exitCode=0 Oct 31 04:19:56.002936 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.002914 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" event={"ID":"aaf851b6-c782-4965-a644-a5e4edfdb82a","Type":"ContainerDied","Data":"70b26b6495ea52f275224b4dc96dd031780bea5a1dc5fc00f5d53553e75d4500"} Oct 31 04:19:56.003477 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.003333 4488 scope.go:122] "RemoveContainer" containerID="70b26b6495ea52f275224b4dc96dd031780bea5a1dc5fc00f5d53553e75d4500" Oct 31 04:19:56.004618 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.004591 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 04:19:56.004699 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.004658 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 04:19:56.004699 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.004693 4488 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 04:19:56.887919 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:56.887859 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:19:57.031718 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:57.031684 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" event={"ID":"aaf851b6-c782-4965-a644-a5e4edfdb82a","Type":"ContainerStarted","Data":"a6d1f09f710d57296f059dbe55608bb9758e173eb1aba263ab75bb6c84931b66"} Oct 31 04:19:57.834464 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:57.834424 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:19:57.835547 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:57.835520 4488 patch_prober.go:29] interesting pod/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:19:57.835674 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:57.835578 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 04:19:58.835296 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:58.835256 4488 patch_prober.go:29] interesting pod/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:19:58.835720 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:58.835314 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 04:19:59.834671 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:59.834631 4488 patch_prober.go:29] interesting pod/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:19:59.834876 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:19:59.834689 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 04:20:00.834676 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:00.834635 4488 patch_prober.go:29] interesting pod/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:20:00.835149 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:00.834696 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 04:20:00.866521 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:00.866487 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:20:01.834532 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:01.834491 4488 patch_prober.go:29] interesting pod/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 31 04:20:01.834735 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:01.834569 4488 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 31 04:20:02.049455 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:02.049426 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:02.838331 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:02.838300 4488 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:20:02.839464 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:02.839441 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:20:06.588655 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:20:06.588613 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:20:06.873317 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:06.873230 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:11.681533 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:11.681501 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:16.523680 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:16.523595 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:20.792876 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:20.792841 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:20.950780 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:20.950724 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:21.389338 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:21.389304 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:25.616562 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:25.616526 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:27.122852 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:27.122815 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:27.406559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:27.406468 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:30.009970 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:30.009933 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx"] Oct 31 04:20:30.010548 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:30.010217 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" containerID="cri-o://a6d1f09f710d57296f059dbe55608bb9758e173eb1aba263ab75bb6c84931b66" gracePeriod=30 Oct 31 04:20:30.011910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:30.011880 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:20:30.025776 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:30.025750 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:31.345704 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:31.345667 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:31.974000 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:31.973968 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:32.143770 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:32.143737 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:32.841891 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:32.841858 4488 patch_prober.go:29] interesting pod/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:20:32.842314 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:32.841911 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:20:35.221015 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.220985 4488 generic.go:356] "Generic (PLEG): container finished" podID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerID="a6d1f09f710d57296f059dbe55608bb9758e173eb1aba263ab75bb6c84931b66" exitCode=0 Oct 31 04:20:35.221541 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.221131 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" event={"ID":"aaf851b6-c782-4965-a644-a5e4edfdb82a","Type":"ContainerDied","Data":"a6d1f09f710d57296f059dbe55608bb9758e173eb1aba263ab75bb6c84931b66"} Oct 31 04:20:35.221541 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.221178 4488 scope.go:122] "RemoveContainer" containerID="70b26b6495ea52f275224b4dc96dd031780bea5a1dc5fc00f5d53553e75d4500" Oct 31 04:20:35.270407 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.270383 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:20:35.376944 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.376910 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.376962 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.376998 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377024 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377132 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377187 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377242 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377270 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377311 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:20:35.377499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377316 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token\") pod \"aaf851b6-c782-4965-a644-a5e4edfdb82a\" (UID: \"aaf851b6-c782-4965-a644-a5e4edfdb82a\") " Oct 31 04:20:35.377499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377414 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:20:35.377499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377489 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:20:35.377813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377564 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:20:35.377813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377518 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:20:35.377813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377800 4488 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.377948 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377819 4488 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-workload-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.377948 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377835 4488 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-credential-socket\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.377948 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377850 4488 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf851b6-c782-4965-a644-a5e4edfdb82a-istiod-ca-cert\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.377948 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.377865 4488 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-data\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.379640 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.379620 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:35.379726 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.379662 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "kube-api-access-2cml9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:20:35.380167 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.380141 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:20:35.380279 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.380184 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo" pod "aaf851b6-c782-4965-a644-a5e4edfdb82a" (UID: "aaf851b6-c782-4965-a644-a5e4edfdb82a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:20:35.478751 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.478713 4488 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-podinfo\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.478751 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.478745 4488 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-envoy\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.478751 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.478755 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cml9\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-kube-api-access-2cml9\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:35.479009 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:35.478766 4488 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf851b6-c782-4965-a644-a5e4edfdb82a-istio-token\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:20:36.118279 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.118239 4488 scope.go:122] "RemoveContainer" containerID="25ad6b666d44b7ca3f7d348d372295f45434e5738b6210a24d1395b45a258670" Oct 31 04:20:36.127654 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.127634 4488 scope.go:122] "RemoveContainer" containerID="ea9932716bacc6b3471a8d3d0ccf874268232d3ff7f9993cda3992d2c00273d4" Oct 31 04:20:36.226997 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.226967 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" Oct 31 04:20:36.227499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.226968 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx" event={"ID":"aaf851b6-c782-4965-a644-a5e4edfdb82a","Type":"ContainerDied","Data":"72a30c7b909c3990bfd2b3e6ed5195be9c6f3220259ed5495d9eb3831e39e054"} Oct 31 04:20:36.227499 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.227198 4488 scope.go:122] "RemoveContainer" containerID="a6d1f09f710d57296f059dbe55608bb9758e173eb1aba263ab75bb6c84931b66" Oct 31 04:20:36.228109 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.228056 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:36.401065 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.400979 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:36.446485 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.446446 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx"] Oct 31 04:20:36.450679 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.450657 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a058534b-openshift-default-9657f649c-xm8rx"] Oct 31 04:20:36.586288 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:36.586250 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:36.589678 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:20:36.589651 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:20:38.055022 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:38.054979 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:38.315938 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:38.315857 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" path="/var/lib/kubelet/pods/aaf851b6-c782-4965-a644-a5e4edfdb82a/volumes" Oct 31 04:20:39.475047 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:39.475006 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:39.651792 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:39.651750 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:42.362655 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.362620 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls"] Oct 31 04:20:42.363122 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.363010 4488 cpu_manager.go:414] "containerMap: removing container" podUID="6a587550-4eaf-4494-832c-ec8d59e692ad" containerName="maas-api" Oct 31 04:20:42.363122 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.363022 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a587550-4eaf-4494-832c-ec8d59e692ad" containerName="maas-api" Oct 31 04:20:42.363122 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.363033 4488 cpu_manager.go:414] "containerMap: removing container" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" Oct 31 04:20:42.363122 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.363041 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" Oct 31 04:20:42.363288 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.363128 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a587550-4eaf-4494-832c-ec8d59e692ad" containerName="maas-api" Oct 31 04:20:42.363288 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.363141 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" Oct 31 04:20:42.366422 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.366405 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.418736 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.418703 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-0bda5d-serving-cert\"" Oct 31 04:20:42.448366 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.448332 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs89s\" (UniqueName: \"kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.448572 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.448390 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.448572 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.448557 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.448704 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.448595 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.470628 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.470591 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls"] Oct 31 04:20:42.549361 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.549324 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.549590 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.549369 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.549590 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:20:42.549495 4488 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-serving-cert: secret "maas-api-e2e-authcfg-http-0bda5d-serving-cert" not found Oct 31 04:20:42.549590 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.549499 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs89s\" (UniqueName: \"kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.549590 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:20:42.549580 4488 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls podName:dc04cb1b-94d6-4531-bb81-bbfa625f9e8c nodeName:}" failed. No retries permitted until 2026-10-31 04:20:43.049559607 +0000 UTC m=+3670.846277871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls") pod "maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" (UID: "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c") : secret "maas-api-e2e-authcfg-http-0bda5d-serving-cert" not found Oct 31 04:20:42.549820 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.549640 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.550257 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.550241 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.551916 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.551900 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.561874 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.561847 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:42.611349 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.611321 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs89s\" (UniqueName: \"kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:42.717545 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.717464 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:42.874083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:42.874048 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:43.055163 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:43.055063 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:43.057852 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:43.057823 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:43.279004 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:43.278968 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:43.654660 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:43.654632 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls"] Oct 31 04:20:43.657137 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:20:43.657079 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc04cb1b_94d6_4531_bb81_bbfa625f9e8c.slice/crio-d068c71d2edb26b1e65e4406696b33ac06a5952e563f5331dd7423221ec3adfb WatchSource:0}: Error finding container d068c71d2edb26b1e65e4406696b33ac06a5952e563f5331dd7423221ec3adfb: Status 404 returned error can't find the container with id d068c71d2edb26b1e65e4406696b33ac06a5952e563f5331dd7423221ec3adfb Oct 31 04:20:44.264675 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:44.264644 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" event={"ID":"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c","Type":"ContainerStarted","Data":"d068c71d2edb26b1e65e4406696b33ac06a5952e563f5331dd7423221ec3adfb"} Oct 31 04:20:45.271330 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:45.271293 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" event={"ID":"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c","Type":"ContainerStarted","Data":"824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57"} Oct 31 04:20:45.271820 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:45.271389 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:45.442569 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:45.442517 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" podStartSLOduration=2.382807803 podStartE2EDuration="3.442504421s" totalImagesPullingTime="1.059696618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:45.440526294 +0000 UTC m=+3673.237244555" watchObservedRunningTime="2026-10-31 04:20:45.442504421 +0000 UTC m=+3673.239222679" Oct 31 04:20:47.611916 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:47.611883 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:50.574638 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:50.574595 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:20:51.281419 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:51.281390 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:20:52.521641 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:52.521609 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:20:57.501083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:20:57.501050 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:01.120687 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:01.120649 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:02.863850 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:02.863810 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:04.185543 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:04.185512 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:06.587417 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:21:06.587376 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:21:07.609582 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:07.609539 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:12.474472 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:12.474438 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:17.290891 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:17.290854 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:22.181960 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:22.181926 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:26.992817 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:26.992776 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:31.881249 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:31.881197 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:36.589507 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:21:36.589458 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:21:36.690174 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:36.690141 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:38.377310 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:38.377272 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:41.508021 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:41.507984 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:46.288903 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:46.288863 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:51.184482 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:51.184446 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:21:55.988800 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:21:55.988764 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:00.881813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:00.881778 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:05.679431 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:05.679395 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:06.250887 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:06.250841 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:06.589260 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:22:06.589225 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:22:10.517812 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:10.517778 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:15.303468 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:15.303427 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:18.713720 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:18.713682 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:20.184211 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:20.184170 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:25.375813 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:25.375771 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:30.377341 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:30.377310 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:33.682191 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:33.682155 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:33.836128 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:33.836075 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:35.194301 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:35.194266 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:36.588099 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:22:36.588050 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:22:38.731998 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:38.731947 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:44.382335 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.382296 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls"] Oct 31 04:22:44.383022 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.382515 4488 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" podUID="dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" containerName="maas-api" containerID="cri-o://824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57" gracePeriod=30 Oct 31 04:22:44.509819 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.509786 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:44.640143 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.640057 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:22:44.723156 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.723111 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\") pod \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " Oct 31 04:22:44.723384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.723168 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca\") pod \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " Oct 31 04:22:44.723384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.723279 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs\") pod \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " Oct 31 04:22:44.723384 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.723341 4488 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s\" (UniqueName: \"kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s\") pod \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\" (UID: \"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c\") " Oct 31 04:22:44.723581 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.723558 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca" pod "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" (UID: "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:22:44.723665 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.723650 4488 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-openshift-service-ca\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:44.725616 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.725590 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls" pod "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" (UID: "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:44.725616 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.725619 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs" pod "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" (UID: "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:22:44.725775 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.725735 4488 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s" pod "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" (UID: "dc04cb1b-94d6-4531-bb81-bbfa625f9e8c"). InnerVolumeSpecName "kube-api-access-qs89s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:22:44.823780 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.823749 4488 generic.go:356] "Generic (PLEG): container finished" podID="dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" containerID="824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57" exitCode=0 Oct 31 04:22:44.823995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.823816 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" event={"ID":"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c","Type":"ContainerDied","Data":"824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57"} Oct 31 04:22:44.823995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.823864 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" event={"ID":"dc04cb1b-94d6-4531-bb81-bbfa625f9e8c","Type":"ContainerDied","Data":"d068c71d2edb26b1e65e4406696b33ac06a5952e563f5331dd7423221ec3adfb"} Oct 31 04:22:44.823995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.823885 4488 scope.go:122] "RemoveContainer" containerID="824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57" Oct 31 04:22:44.823995 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.823817 4488 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls" Oct 31 04:22:44.824361 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.824343 4488 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-metrics-certs\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:44.824431 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.824368 4488 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs89s\" (UniqueName: \"kubernetes.io/projected/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-kube-api-access-qs89s\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:44.824431 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.824382 4488 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c-maas-api-tls\") on node \"ip-10-0-143-24.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 04:22:44.834241 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.834222 4488 scope.go:122] "RemoveContainer" containerID="824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57" Oct 31 04:22:44.834529 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:22:44.834509 4488 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57\": container with ID starting with 824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57 not found: ID does not exist" containerID="824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57" Oct 31 04:22:44.834599 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.834541 4488 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57"} err="failed to get container status \"824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57\": rpc error: code = NotFound desc = could not find container \"824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57\": container with ID starting with 824e5a7adad4d409979d2f8c90fabc2b25bac32e416814edb6a6a9efcc8f8a57 not found: ID does not exist" Oct 31 04:22:44.988419 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.988321 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls"] Oct 31 04:22:44.992474 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:44.992440 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0bda5d-788bf7cff6-8plls"] Oct 31 04:22:46.011822 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:46.011783 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:46.294252 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:46.294159 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:46.315183 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:46.315151 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" path="/var/lib/kubelet/pods/dc04cb1b-94d6-4531-bb81-bbfa625f9e8c/volumes" Oct 31 04:22:47.124951 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:47.124920 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:49.519666 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:49.519620 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:22:53.238629 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:53.238581 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:53.397174 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:53.397072 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:22:53.595514 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:22:53.595483 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:23:01.193770 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:01.193732 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:23:06.588221 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:23:06.588184 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:23:12.052488 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:12.052445 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4"] Oct 31 04:23:12.102165 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:12.102128 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5wqs4"] Oct 31 04:23:12.318671 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:12.318592 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19031f15-7bc1-4377-9730-a3368593113d" path="/var/lib/kubelet/pods/19031f15-7bc1-4377-9730-a3368593113d/volumes" Oct 31 04:23:12.888914 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:12.888875 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:23:33.966243 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:33.966204 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:23:35.644851 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:35.644815 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:23:35.857387 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:35.857344 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:23:36.588161 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:23:36.588120 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:23:39.119722 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:39.119687 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:23:41.057176 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.057123 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq"] Oct 31 04:23:41.101703 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.101644 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fgnqq"] Oct 31 04:23:41.187821 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.187199 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-xxm8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:23:41.187821 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.187283 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-xxm8g" podUID="ee63cc13-5d3f-4f87-a8f8-2c7987300722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:23:41.227239 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.227200 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-kqkqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:23:41.227532 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.227501 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-kqkqf" podUID="1a068c85-58af-48e3-9fe0-55756d21fb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:23:41.615321 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.615283 4488 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-6d674 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:23:41.615657 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:41.615622 4488 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-6d674" podUID="eaf69280-a762-4868-9057-76241c023adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:23:42.315175 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:23:42.315143 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2510e7d2-caea-4c22-b8ca-152f99864799" path="/var/lib/kubelet/pods/2510e7d2-caea-4c22-b8ca-152f99864799/volumes" Oct 31 04:24:05.971198 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:05.971163 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6746c6f78c-5kp45_d833b50c-4425-47f6-b1cc-f1dfb5154578/authorino/0.log" Oct 31 04:24:06.588596 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:24:06.588560 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:24:06.706232 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:06.706195 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gc8k4"] Oct 31 04:24:12.046649 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:12.046608 4488 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852"] Oct 31 04:24:12.096412 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:12.096381 4488 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c072c2-wr852"] Oct 31 04:24:12.315319 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:12.315240 4488 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f50f9439-8dd3-40f0-8393-acb83b97e24b" path="/var/lib/kubelet/pods/f50f9439-8dd3-40f0-8393-acb83b97e24b/volumes" Oct 31 04:24:19.956997 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:19.956963 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796fbdf5b4-ds97c_f05dd293-d64e-4dac-910f-27f6569ae93f/maas-api/0.log" Oct 31 04:24:22.717323 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:22.717286 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6746c6f78c-5kp45_d833b50c-4425-47f6-b1cc-f1dfb5154578/authorino/0.log" Oct 31 04:24:24.410609 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:24.410581 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gc8k4_9ef7c187-78c1-47cd-86e6-4c4af998f3ba/limitador/0.log" Oct 31 04:24:25.893924 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:25.893896 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-764498d796-5l9zv_f4ac29c7-2389-44ce-90af-a241e3e98271/kube-auth-proxy/0.log" Oct 31 04:24:26.345823 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:26.345797 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-6d674_eaf69280-a762-4868-9057-76241c023adf/istio-proxy/0.log" Oct 31 04:24:27.324343 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:27.324317 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-kqkqf_1a068c85-58af-48e3-9fe0-55756d21fb67/istio-proxy/0.log" Oct 31 04:24:28.334973 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:28.334933 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-xxm8g_ee63cc13-5d3f-4f87-a8f8-2c7987300722/istio-proxy/0.log" Oct 31 04:24:28.610559 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:28.610478 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-znzrz_7f7842f2-5eeb-417d-8bcb-76f99f23fe23/istio-proxy/0.log" Oct 31 04:24:29.065462 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:29.065385 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zqpt2_f5ba0a01-b535-458e-b03b-63a5f48eeaeb/payload-pre-processing/0.log" Oct 31 04:24:31.976270 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:31.976234 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d_e252476b-aafe-4d02-9673-c55bbb915a52/main/0.log" Oct 31 04:24:33.032872 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:33.032836 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6746c6f78c-5kp45_d833b50c-4425-47f6-b1cc-f1dfb5154578/authorino/0.log" Oct 31 04:24:36.185720 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:36.185684 4488 scope.go:122] "RemoveContainer" containerID="6e6f5dc64f81a8e3f5ebae0a6ba296310af038c134d5367d526d7d3f0cfcbce3" Oct 31 04:24:36.589022 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:24:36.588974 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:24:39.432234 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:39.432201 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:24:40.103920 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:40.103889 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:24:44.832573 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:44.832494 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796fbdf5b4-ds97c_f05dd293-d64e-4dac-910f-27f6569ae93f/maas-api/0.log" Oct 31 04:24:47.332278 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:47.332247 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6746c6f78c-5kp45_d833b50c-4425-47f6-b1cc-f1dfb5154578/authorino/0.log" Oct 31 04:24:48.966787 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:48.966753 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gc8k4_9ef7c187-78c1-47cd-86e6-4c4af998f3ba/limitador/0.log" Oct 31 04:24:50.270442 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:50.270409 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-764498d796-5l9zv_f4ac29c7-2389-44ce-90af-a241e3e98271/kube-auth-proxy/0.log" Oct 31 04:24:50.728240 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:50.728208 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-6d674_eaf69280-a762-4868-9057-76241c023adf/istio-proxy/0.log" Oct 31 04:24:51.754080 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:51.754044 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-kqkqf_1a068c85-58af-48e3-9fe0-55756d21fb67/istio-proxy/0.log" Oct 31 04:24:52.783918 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:52.783890 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-xxm8g_ee63cc13-5d3f-4f87-a8f8-2c7987300722/istio-proxy/0.log" Oct 31 04:24:53.069318 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:53.069235 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-znzrz_7f7842f2-5eeb-417d-8bcb-76f99f23fe23/istio-proxy/0.log" Oct 31 04:24:53.525777 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:53.525749 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zqpt2_f5ba0a01-b535-458e-b03b-63a5f48eeaeb/payload-pre-processing/0.log" Oct 31 04:24:56.280779 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:24:56.280750 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dv27d_e252476b-aafe-4d02-9673-c55bbb915a52/main/0.log" Oct 31 04:25:06.589507 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:25:06.589472 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:25:15.773747 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:25:15.773713 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:25:36.209060 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:25:36.209023 4488 scope.go:122] "RemoveContainer" containerID="da33c79cfd23f76d3b90890f3a53f5b3146d5cea7d3c29fbedcc44a8a12ce2dc" Oct 31 04:25:36.217543 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:25:36.217522 4488 scope.go:122] "RemoveContainer" containerID="229e7bb987450036ee296a08f7434d2be67a051c1ea45923b30ffac4fdef03e3" Oct 31 04:25:36.589481 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:25:36.589434 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:25:39.742600 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:25:39.742568 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:26:06.588125 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:26:06.588059 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:26:08.690979 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:08.690943 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:26:08.852436 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:08.852393 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:26:18.881513 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:18.881485 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jvb8m_d66b1354-90f5-43ec-b395-cabd3dc171ac/global-pull-secret-syncer/0.log" Oct 31 04:26:20.837691 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:20.837653 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" Oct 31 04:26:23.036597 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:23.036563 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-24.us-west-1.compute.internal_abb3780775d238ba0d694f1e9cc7fb2a/haproxy/0.log" Oct 31 04:26:27.249232 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.249194 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6746c6f78c-5kp45_d833b50c-4425-47f6-b1cc-f1dfb5154578/authorino/0.log" Oct 31 04:26:27.564800 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.564712 4488 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt"] Oct 31 04:26:27.565080 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.565068 4488 cpu_manager.go:414] "containerMap: removing container" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" Oct 31 04:26:27.565160 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.565081 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" Oct 31 04:26:27.565160 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.565149 4488 cpu_manager.go:414] "containerMap: removing container" podUID="dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" containerName="maas-api" Oct 31 04:26:27.565160 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.565154 4488 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" containerName="maas-api" Oct 31 04:26:27.565314 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.565220 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc04cb1b-94d6-4531-bb81-bbfa625f9e8c" containerName="maas-api" Oct 31 04:26:27.565314 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.565232 4488 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf851b6-c782-4965-a644-a5e4edfdb82a" containerName="istio-proxy" Oct 31 04:26:27.568394 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.568376 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.625512 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.625485 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr5kx\"/\"kube-root-ca.crt\"" Oct 31 04:26:27.675083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.675050 4488 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr5kx\"/\"openshift-service-ca.crt\"" Oct 31 04:26:27.675083 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.675051 4488 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xr5kx\"/\"default-dockercfg-mqfbg\"" Oct 31 04:26:27.677074 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.677049 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt"] Oct 31 04:26:27.681652 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.681631 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-lib-modules\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.681779 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.681658 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-sys\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.681779 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.681673 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-podres\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.681779 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.681707 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-proc\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.681914 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.681866 4488 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5ml\" (UniqueName: \"kubernetes.io/projected/306e2a95-f3c4-422a-8ae1-09221be64d2a-kube-api-access-kd5ml\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782523 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782489 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-proc\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782555 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5ml\" (UniqueName: \"kubernetes.io/projected/306e2a95-f3c4-422a-8ae1-09221be64d2a-kube-api-access-kd5ml\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782619 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-proc\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782661 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-lib-modules\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782690 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-sys\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782713 4488 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-podres\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782725 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-lib-modules\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.782758 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782745 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-sys\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.783056 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.782837 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/306e2a95-f3c4-422a-8ae1-09221be64d2a-podres\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.843910 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.843865 4488 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5ml\" (UniqueName: \"kubernetes.io/projected/306e2a95-f3c4-422a-8ae1-09221be64d2a-kube-api-access-kd5ml\") pod \"perf-node-gather-daemonset-8dzgt\" (UID: \"306e2a95-f3c4-422a-8ae1-09221be64d2a\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:27.881231 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:27.881188 4488 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:28.255033 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:28.255004 4488 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt"] Oct 31 04:26:28.257761 ip-10-0-143-24 kubenswrapper[4488]: W1031 04:26:28.257732 4488 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod306e2a95_f3c4_422a_8ae1_09221be64d2a.slice/crio-762beb20c06d15f23a83527fabe45add163cb22b624b10f6fe11d344f97fdd9a WatchSource:0}: Error finding container 762beb20c06d15f23a83527fabe45add163cb22b624b10f6fe11d344f97fdd9a: Status 404 returned error can't find the container with id 762beb20c06d15f23a83527fabe45add163cb22b624b10f6fe11d344f97fdd9a Oct 31 04:26:28.799861 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:28.799819 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:26:28.946979 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:28.946944 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" event={"ID":"306e2a95-f3c4-422a-8ae1-09221be64d2a","Type":"ContainerStarted","Data":"b132ec3b9c304a45ba1d1b485d4b36ec6bcfaf3dce0d00acd969d92de5098294"} Oct 31 04:26:28.947204 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:28.946995 4488 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" event={"ID":"306e2a95-f3c4-422a-8ae1-09221be64d2a","Type":"ContainerStarted","Data":"762beb20c06d15f23a83527fabe45add163cb22b624b10f6fe11d344f97fdd9a"} Oct 31 04:26:28.947204 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:28.947106 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:29.065872 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:29.065774 4488 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" podStartSLOduration=2.06575524 podStartE2EDuration="2.06575524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:29.064056098 +0000 UTC m=+4016.860774357" watchObservedRunningTime="2026-10-31 04:26:29.06575524 +0000 UTC m=+4016.862473497" Oct 31 04:26:29.364740 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:29.364711 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gc8k4_9ef7c187-78c1-47cd-86e6-4c4af998f3ba/limitador/0.log" Oct 31 04:26:34.962619 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:34.962589 4488 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-8dzgt" Oct 31 04:26:36.588386 ip-10-0-143-24 kubenswrapper[4488]: E1031 04:26:36.588350 4488 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qs77w" containerName="extractor" Oct 31 04:26:37.896448 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:37.896414 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2m89j_02356a1a-5fe3-424d-95b3-7d3e6f7e9b06/node-exporter/0.log" Oct 31 04:26:38.314773 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:38.314688 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2m89j_02356a1a-5fe3-424d-95b3-7d3e6f7e9b06/kube-rbac-proxy/0.log" Oct 31 04:26:38.749210 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:38.749178 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2m89j_02356a1a-5fe3-424d-95b3-7d3e6f7e9b06/init-textfile/0.log" Oct 31 04:26:40.125738 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:40.125704 4488 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms5j_ae4c6a55-1343-49ab-b514-78564e0f1d75/konnectivity-agent/0.log" limit=1024 Oct 31 04:26:41.720843 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:41.720815 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x9b6w_6b2d2183-b5ee-476a-9085-4e01563d8f13/kube-rbac-proxy-main/0.log" Oct 31 04:26:42.138470 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:42.138440 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x9b6w_6b2d2183-b5ee-476a-9085-4e01563d8f13/kube-rbac-proxy-self/0.log" Oct 31 04:26:42.556997 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:42.556890 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x9b6w_6b2d2183-b5ee-476a-9085-4e01563d8f13/openshift-state-metrics/0.log" Oct 31 04:26:42.990419 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:42.990391 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/prometheus/0.log" Oct 31 04:26:43.409661 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:43.409635 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/config-reloader/0.log" Oct 31 04:26:43.833470 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:43.833446 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/thanos-sidecar/0.log" Oct 31 04:26:44.265295 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:44.265221 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/kube-rbac-proxy-web/0.log" Oct 31 04:26:44.683648 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:44.683616 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/kube-rbac-proxy/0.log" Oct 31 04:26:45.101727 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:45.101700 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/kube-rbac-proxy-thanos/0.log" Oct 31 04:26:45.541263 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:45.541186 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c3afbd4-990d-4870-87b5-f05c511517c0/init-config-reloader/0.log" Oct 31 04:26:46.815375 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:46.815344 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7lhsx_3a69464e-a1dc-4a52-a80a-a53c65aafde5/prometheus-operator-admission-webhook/0.log" Oct 31 04:26:51.665390 ip-10-0-143-24 kubenswrapper[4488]: I1031 04:26:51.665365 4488 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-764498d796-5l9zv_f4ac29c7-2389-44ce-90af-a241e3e98271/kube-auth-proxy/0.log"